Scrape Content
Web Scraping
Scrape Content
Create a new scraping session to extract content from any website with optional AI-powered data extraction
POST
Scrape Content
Overview
The/scrape endpoint initiates a new web scraping job with the specified URL and options. It returns the scraped content directly and supports both renderless and renderful scraping modes. You can also use AI-powered extraction to get structured data from the page.
Parameters
Request Body
string
required
The URL to scrape
boolean
default:"false"
Whether the request should be asynchronous. When
true, returns a job ID immediately that can be used to check status.boolean
default:"false"
Whether to render JavaScript on the page. Enable for dynamic content loaded via JavaScript.
string
default:"get"
HTTP method for the request. Options:
get, post, put, delete, patch, head, optionsstring
default:"raw"
Response format. Options:
raw, json, markdown.When format is markdown, ScrapEngine auto-detects the response type:- HTML → converted via Turndown
- PDF / XLSX / DOCX → extracted via the built-in document parser (see
documentOptionsbelow)
string
default:"us"
Proxy country for geo-targeted requests (e.g.,
us, uk, de)boolean
default:"false"
Whether to include response headers in the response
object
HTTP headers for the request
object
Request body content (for POST/PUT requests)
Document Extraction Options
Applied only whenformat is markdown AND the target response is a PDF, XLSX, or DOCX file. Ignored for HTML responses.
object
PDF / XLSX / DOCX extraction options.
string
Password for encrypted PDFs.
number[]
1-based page numbers to include (PDF only). Omit for the whole document.
string
How PDF page boundaries appear in the output markdown. Options:
none (no separator), hr (horizontal rule), comment (HTML comment).Strip recurring page headers and footers from PDF output.
LLM Extraction Options
object
AI-powered extraction options. When provided, the scraped content will be processed by an LLM to extract structured data.
object
JSON Schema defining the structure to extract. Use this for precise, typed extraction.
string
Natural language prompt describing what to extract. Use this for flexible, conversational extraction.
string
Custom system prompt to guide the LLM behavior
string
default:"gpt-4o-mini"
LLM model to use for extraction. Options:
gpt-4o, gpt-4o-mini, claude-3-5-sonnetboolean
default:"true"
Whether to include extraction metadata (tokens used, cost) in response
Example Requests
Basic Scraping
Scrape a PDF as Markdown
Point/scrape at a PDF (or XLSX / DOCX) URL with format: "markdown" to get extracted text back as Markdown.
With LLM Extraction (Schema)
Extract structured data using a JSON schema:With LLM Extraction (Prompt)
Extract data using a natural language prompt:Response
Success Response (200)
Without extraction - Returns the scraped HTML content directly:Response Headers
Error Responses
Error Response Format:
Use Cases
- E-commerce scraping: Extract product information, prices, and availability
- Content aggregation: Collect articles, blog posts, and news content
- Lead generation: Extract contact information and company details
- Competitor analysis: Monitor competitor websites and pricing
- SEO analysis: Extract meta tags, headings, and content structure
- AI-powered extraction: Use LLM to extract structured data without writing parsers