Skip to main content
POST
Scrape Content

Overview

The /scrape endpoint initiates a new web scraping job with the specified URL and options. It returns the scraped content directly and supports both renderless and renderful scraping modes. You can also use AI-powered extraction to get structured data from the page.

Parameters

Request Body

string
required
The URL to scrape
boolean
default:"false"
Whether the request should be asynchronous. When true, returns a job ID immediately that can be used to check status.
boolean
default:"false"
Whether to render JavaScript on the page. Enable for dynamic content loaded via JavaScript.
string
default:"get"
HTTP method for the request. Options: get, post, put, delete, patch, head, options
string
default:"raw"
Response format. Options: raw, json, markdown.When format is markdown, ScrapEngine auto-detects the response type:
  • HTML → converted via Turndown
  • PDF / XLSX / DOCX → extracted via the built-in document parser (see documentOptions below)
string
default:"us"
Proxy country for geo-targeted requests (e.g., us, uk, de)
boolean
default:"false"
Whether to include response headers in the response
object
HTTP headers for the request
object
Request body content (for POST/PUT requests)

Document Extraction Options

Applied only when format is markdown AND the target response is a PDF, XLSX, or DOCX file. Ignored for HTML responses.
object
PDF / XLSX / DOCX extraction options.
string
Password for encrypted PDFs.
number[]
1-based page numbers to include (PDF only). Omit for the whole document.
string
How PDF page boundaries appear in the output markdown. Options: none (no separator), hr (horizontal rule), comment (HTML comment).
boolean
default:"true"
Strip recurring page headers and footers from PDF output.

LLM Extraction Options

object
AI-powered extraction options. When provided, the scraped content will be processed by an LLM to extract structured data.
object
JSON Schema defining the structure to extract. Use this for precise, typed extraction.
string
Natural language prompt describing what to extract. Use this for flexible, conversational extraction.
string
Custom system prompt to guide the LLM behavior
string
default:"gpt-4o-mini"
LLM model to use for extraction. Options: gpt-4o, gpt-4o-mini, claude-3-5-sonnet
boolean
default:"true"
Whether to include extraction metadata (tokens used, cost) in response

Example Requests

Basic Scraping

Scrape a PDF as Markdown

Point /scrape at a PDF (or XLSX / DOCX) URL with format: "markdown" to get extracted text back as Markdown.

With LLM Extraction (Schema)

Extract structured data using a JSON schema:

With LLM Extraction (Prompt)

Extract data using a natural language prompt:

Response

Success Response (200)

Without extraction - Returns the scraped HTML content directly:
With extraction - Returns structured JSON:

Response Headers

Error Responses

Error Response Format:

Use Cases

  • E-commerce scraping: Extract product information, prices, and availability
  • Content aggregation: Collect articles, blog posts, and news content
  • Lead generation: Extract contact information and company details
  • Competitor analysis: Monitor competitor websites and pricing
  • SEO analysis: Extract meta tags, headings, and content structure
  • AI-powered extraction: Use LLM to extract structured data without writing parsers