> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapengine.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Content

> Create a new scraping session to extract content from any website with optional AI-powered data extraction

## Overview

The `/scrape` endpoint initiates a new web scraping job with the specified URL and options. It returns the scraped content directly and supports both renderless and renderful scraping modes. You can also use AI-powered extraction to get structured data from the page.

## Parameters

### Request Body

<ParamField body="url" type="string" required>
  The URL to scrape
</ParamField>

<ParamField body="async" type="boolean" default="false">
  Whether the request should be asynchronous. When `true`, returns a job ID immediately that can be used to check status.
</ParamField>

<ParamField body="render" type="boolean" default="false">
  Whether to render JavaScript on the page. Enable for dynamic content loaded via JavaScript.
</ParamField>

<ParamField body="method" type="string" default="get">
  HTTP method for the request. Options: `get`, `post`, `put`, `delete`, `patch`, `head`, `options`
</ParamField>

<ParamField body="format" type="string" default="raw">
  Response format. Options: `raw`, `json`, `markdown`.

  When `format` is `markdown`, ScrapEngine auto-detects the response type:

  * **HTML** → converted via Turndown
  * **PDF / XLSX / DOCX** → extracted via the built-in document parser (see `documentOptions` below)
</ParamField>

<ParamField body="country" type="string" default="us">
  Proxy country for geo-targeted requests (e.g., `us`, `uk`, `de`)
</ParamField>

<ParamField body="includeHeaders" type="boolean" default="false">
  Whether to include response headers in the response
</ParamField>

<ParamField body="headers" type="object">
  HTTP headers for the request

  ```json theme={null}
  {
    "User-Agent": "Mozilla/5.0...",
    "Accept-Language": "en-US,en;q=0.9"
  }
  ```
</ParamField>

<ParamField body="body" type="object">
  Request body content (for POST/PUT requests)
</ParamField>

### Document Extraction Options

Applied only when `format` is `markdown` AND the target response is a PDF, XLSX, or DOCX file. Ignored for HTML responses.

<ParamField body="documentOptions" type="object">
  PDF / XLSX / DOCX extraction options.
</ParamField>

<ParamField body="documentOptions.password" type="string">
  Password for encrypted PDFs.
</ParamField>

<ParamField body="documentOptions.pages" type="number[]">
  1-based page numbers to include (PDF only). Omit for the whole document.

  ```json theme={null}
  { "pages": [1, 2, 5] }
  ```
</ParamField>

<ParamField body="documentOptions.pageBreak" type="string">
  How PDF page boundaries appear in the output markdown. Options: `none` (no separator), `hr` (horizontal rule), `comment` (HTML comment).
</ParamField>

<ParamField body="documentOptions.stripHeadersFooters" type="boolean" default="true">
  Strip recurring page headers and footers from PDF output.
</ParamField>

### LLM Extraction Options

<ParamField body="extract" type="object">
  AI-powered extraction options. When provided, the scraped content will be processed by an LLM to extract structured data.
</ParamField>

<ParamField body="extract.schema" type="object">
  JSON Schema defining the structure to extract. Use this for precise, typed extraction.

  ```json theme={null}
  {
    "type": "object",
    "properties": {
      "title": { "type": "string", "description": "The page title" },
      "price": { "type": "number", "description": "Product price" },
      "features": {
        "type": "array",
        "items": { "type": "string" }
      }
    },
    "required": ["title", "price"]
  }
  ```
</ParamField>

<ParamField body="extract.prompt" type="string">
  Natural language prompt describing what to extract. Use this for flexible, conversational extraction.
</ParamField>

<ParamField body="extract.systemPrompt" type="string">
  Custom system prompt to guide the LLM behavior
</ParamField>

<ParamField body="extract.model" type="string" default="gpt-4o-mini">
  LLM model to use for extraction. Options: `gpt-4o`, `gpt-4o-mini`, `claude-3-5-sonnet`
</ParamField>

<ParamField body="extract.includeMetadata" type="boolean" default="true">
  Whether to include extraction metadata (tokens used, cost) in response
</ParamField>

## Example Requests

### Basic Scraping

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST "https://api.scrapengine.io/api/v1/scrape" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://www.example.com/product",
      "render": true
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch("https://api.scrapengine.io/api/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: "https://www.example.com/product",
      render: true,
    }),
  });

  const data = await response.text();
  console.log(data);
  ```

  ```python Python theme={null}
  import requests

  url = "https://api.scrapengine.io/api/v1/scrape"
  headers = {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json"
  }
  data = {
      "url": "https://www.example.com/product",
      "render": True
  }

  response = requests.post(url, headers=headers, json=data)
  print(response.text)
  ```
</CodeGroup>

### Scrape a PDF as Markdown

Point `/scrape` at a PDF (or XLSX / DOCX) URL with `format: "markdown"` to get extracted text back as Markdown.

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST "https://api.scrapengine.io/api/v1/scrape" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://www.example.com/whitepaper.pdf",
      "format": "markdown",
      "documentOptions": {
        "pages": [1, 2, 3],
        "stripHeadersFooters": true
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch("https://api.scrapengine.io/api/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: "https://www.example.com/whitepaper.pdf",
      format: "markdown",
      documentOptions: {
        pages: [1, 2, 3],
        stripHeadersFooters: true,
      },
    }),
  });

  const markdown = await response.text();
  console.log(markdown);
  ```

  ```python Python theme={null}
  import requests

  url = "https://api.scrapengine.io/api/v1/scrape"
  headers = {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json"
  }
  data = {
      "url": "https://www.example.com/whitepaper.pdf",
      "format": "markdown",
      "documentOptions": {
          "pages": [1, 2, 3],
          "stripHeadersFooters": True
      }
  }

  response = requests.post(url, headers=headers, json=data)
  print(response.text)
  ```
</CodeGroup>

### With LLM Extraction (Schema)

Extract structured data using a JSON schema:

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST "https://api.scrapengine.io/api/v1/scrape" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://www.example.com/product",
      "render": true,
      "extract": {
        "schema": {
          "type": "object",
          "properties": {
            "productName": { "type": "string", "description": "The product name" },
            "price": { "type": "number", "description": "Price in USD" },
            "rating": { "type": "number", "description": "Average rating out of 5" },
            "features": {
              "type": "array",
              "items": { "type": "string" },
              "description": "List of product features"
            }
          },
          "required": ["productName", "price"]
        },
        "model": "gpt-4o-mini"
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch("https://api.scrapengine.io/api/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: "https://www.example.com/product",
      render: true,
      extract: {
        schema: {
          type: "object",
          properties: {
            productName: { type: "string", description: "The product name" },
            price: { type: "number", description: "Price in USD" },
            rating: { type: "number", description: "Average rating out of 5" },
            features: {
              type: "array",
              items: { type: "string" },
              description: "List of product features",
            },
          },
          required: ["productName", "price"],
        },
        model: "gpt-4o-mini",
      },
    }),
  });

  const data = await response.json();
  console.log(data);
  ```

  ```python Python theme={null}
  import requests

  url = "https://api.scrapengine.io/api/v1/scrape"
  headers = {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json"
  }
  data = {
      "url": "https://www.example.com/product",
      "render": True,
      "extract": {
          "schema": {
              "type": "object",
              "properties": {
                  "productName": {"type": "string", "description": "The product name"},
                  "price": {"type": "number", "description": "Price in USD"},
                  "rating": {"type": "number", "description": "Average rating out of 5"},
                  "features": {
                      "type": "array",
                      "items": {"type": "string"},
                      "description": "List of product features"
                  }
              },
              "required": ["productName", "price"]
          },
          "model": "gpt-4o-mini"
      }
  }

  response = requests.post(url, headers=headers, json=data)
  print(response.json())
  ```
</CodeGroup>

### With LLM Extraction (Prompt)

Extract data using a natural language prompt:

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST "https://api.scrapengine.io/api/v1/scrape" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://www.example.com/article",
      "extract": {
        "prompt": "Extract the article title, author name, publication date, and a brief summary of the main points",
        "model": "claude-3-5-sonnet"
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch("https://api.scrapengine.io/api/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: "https://www.example.com/article",
      extract: {
        prompt:
          "Extract the article title, author name, publication date, and a brief summary of the main points",
        model: "claude-3-5-sonnet",
      },
    }),
  });

  const data = await response.json();
  console.log(data);
  ```

  ```python Python theme={null}
  import requests

  url = "https://api.scrapengine.io/api/v1/scrape"
  headers = {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json"
  }
  data = {
      "url": "https://www.example.com/article",
      "extract": {
          "prompt": "Extract the article title, author name, publication date, and a brief summary of the main points",
          "model": "claude-3-5-sonnet"
      }
  }

  response = requests.post(url, headers=headers, json=data)
  print(response.json())
  ```
</CodeGroup>

## Response

### Success Response (200)

**Without extraction** - Returns the scraped HTML content directly:

```html theme={null}
<html>
  <head>
    <title>Example Product</title>
  </head>
  <body>
    <!-- Scraped content here -->
  </body>
</html>
```

**With extraction** - Returns structured JSON:

```json theme={null}
{
  "data": {
    "productName": "Premium Wireless Headphones",
    "price": 299.99,
    "rating": 4.5,
    "features": [
      "Active Noise Cancellation",
      "40-hour battery life",
      "Bluetooth 5.0"
    ]
  },
  "metadata": {
    "tokensUsed": 1250,
    "model": "gpt-4o-mini"
  }
}
```

### Response Headers

| Header                | Description                       |
| --------------------- | --------------------------------- |
| `x-remaining-credits` | Number of API credits remaining   |
| `x-trace-id`          | Unique identifier for the request |

### Error Responses

| Status | Description                                                |
| ------ | ---------------------------------------------------------- |
| `400`  | Bad Request - Invalid parameters or URL                    |
| `401`  | Unauthorized - Invalid or missing API key                  |
| `403`  | Forbidden - Access denied to target resource               |
| `404`  | Not Found - Target URL not found                           |
| `408`  | Request Timeout                                            |
| `500`  | Internal Server Error                                      |
| `550`  | Faulted After Retries - Job failed after multiple attempts |

**Error Response Format:**

```json theme={null}
{
  "error": "Error description message",
  "traceId": "abc123-def456",
  "timestamp": "2024-01-15T10:30:00.000Z"
}
```

## Use Cases

* **E-commerce scraping**: Extract product information, prices, and availability
* **Content aggregation**: Collect articles, blog posts, and news content
* **Lead generation**: Extract contact information and company details
* **Competitor analysis**: Monitor competitor websites and pricing
* **SEO analysis**: Extract meta tags, headings, and content structure
* **AI-powered extraction**: Use LLM to extract structured data without writing parsers