> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapengine.io/llms.txt
> Use this file to discover all available pages before exploring further.

# HTTP Proxy

> Use Scrapengine as a standard HTTP/HTTPS proxy for seamless integration with existing tools and libraries

## Overview

The Scrapengine Proxy allows you to use Scrapengine as a standard HTTP/HTTPS proxy server. This is ideal for integrating with existing tools, libraries, or applications that support proxy configuration. Simply point your HTTP client to our proxy endpoint and authenticate using your API key.

<Info>
  The proxy interface provides the same scraping capabilities as the REST API but through a standard proxy protocol that works with any HTTP client.
</Info>

## Proxy Endpoint

```
gw.scrapengine.io:8081
```

## Authentication

Authentication is done via HTTP Basic Auth where:

* **Username**: Any value (e.g., `scrape`)
* **Password**: Your Scrapengine API key

## Quick Start

<CodeGroup>
  ```bash cURL theme={null}
  curl -x http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081 \
    https://example.com
  ```

  ```javascript Node.js (axios) theme={null}
  const axios = require('axios');

  const response = await axios.get('https://example.com', {
    proxy: {
      host: 'gw.scrapengine.io',
      port: 8081,
      auth: {
        username: 'scrape',
        password: 'YOUR_API_KEY'
      }
    }
  });

  console.log(response.data);
  ```

  ```python Python (requests) theme={null}
  import requests

  proxies = {
      'http': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081',
      'https': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'
  }

  response = requests.get('https://example.com', proxies=proxies)
  print(response.text)
  ```

  ```python Python (httpx) theme={null}
  import httpx

  proxy = 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'

  with httpx.Client(proxy=proxy) as client:
      response = client.get('https://example.com')
      print(response.text)
  ```

  ```javascript Node.js (node-fetch with proxy-agent) theme={null}
  const fetch = require('node-fetch');
  const { HttpsProxyAgent } = require('https-proxy-agent');

  const agent = new HttpsProxyAgent('http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081');

  const response = await fetch('https://example.com', { agent });
  console.log(await response.text());
  ```

  ```go Go theme={null}
  package main

  import (
      "fmt"
      "io"
      "net/http"
      "net/url"
  )

  func main() {
      proxyURL, _ := url.Parse("http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081")

      client := &http.Client{
          Transport: &http.Transport{
              Proxy: http.ProxyURL(proxyURL),
          },
      }

      resp, _ := client.Get("https://example.com")
      defer resp.Body.Close()

      body, _ := io.ReadAll(resp.Body)
      fmt.Println(string(body))
  }
  ```
</CodeGroup>

## Configuration Headers

Control scraping behavior by passing custom headers with your request. These headers are parsed by the proxy and configure how the scraping job is executed.

| Header                          | Type    | Default | Description                                         |
| ------------------------------- | ------- | ------- | --------------------------------------------------- |
| `x-scrapengine-render`          | boolean | `false` | Enable JavaScript rendering for dynamic content     |
| `x-scrapengine-async`           | boolean | `false` | Return immediately with job ID for async processing |
| `x-scrapengine-location`        | string  | `us`    | Proxy country code (e.g., `us`, `uk`, `de`)         |
| `x-scrapengine-format`          | string  | `raw`   | Response format: `raw`, `json`, or `markdown`       |
| `x-scrapengine-include-headers` | boolean | `false` | Include response headers in the response            |
| `x-scrapengine-verbose`         | boolean | `false` | Return detailed response headers                    |

## Examples with Headers

### Enable JavaScript Rendering

Scrape pages that require JavaScript to load content:

<CodeGroup>
  ```bash cURL theme={null}
  curl -x http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081 \
    -H "x-scrapengine-render: true" \
    https://example.com/dynamic-page
  ```

  ```python Python theme={null}
  import requests

  proxies = {
      'http': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081',
      'https': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'
  }

  headers = {
      'x-scrapengine-render': 'true'
  }

  response = requests.get('https://example.com/dynamic-page', proxies=proxies, headers=headers)
  print(response.text)
  ```

  ```javascript Node.js theme={null}
  const axios = require('axios');

  const response = await axios.get('https://example.com/dynamic-page', {
    proxy: {
      host: 'gw.scrapengine.io',
      port: 8081,
      auth: {
        username: 'scrape',
        password: 'YOUR_API_KEY'
      }
    },
    headers: {
      'x-scrapengine-render': 'true'
    }
  });

  console.log(response.data);
  ```
</CodeGroup>

### Get Markdown Output

Convert the scraped content to clean Markdown:

<CodeGroup>
  ```bash cURL theme={null}
  curl -x http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081 \
    -H "x-scrapengine-format: markdown" \
    https://example.com/article
  ```

  ```python Python theme={null}
  import requests

  proxies = {
      'http': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081',
      'https': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'
  }

  headers = {
      'x-scrapengine-format': 'markdown'
  }

  response = requests.get('https://example.com/article', proxies=proxies, headers=headers)
  print(response.text)
  ```
</CodeGroup>

### Geo-Targeted Scraping

Scrape from a specific geographic location:

<CodeGroup>
  ```bash cURL theme={null}
  curl -x http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081 \
    -H "x-scrapengine-location: uk" \
    -H "x-scrapengine-render: true" \
    https://example.com/local-prices
  ```

  ```python Python theme={null}
  import requests

  proxies = {
      'http': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081',
      'https': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'
  }

  headers = {
      'x-scrapengine-location': 'uk',
      'x-scrapengine-render': 'true'
  }

  response = requests.get('https://example.com/local-prices', proxies=proxies, headers=headers)
  print(response.text)
  ```
</CodeGroup>

### Multiple Options Combined

Combine multiple options for advanced scraping:

<CodeGroup>
  ```bash cURL theme={null}
  curl -x http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081 \
    -H "x-scrapengine-render: true" \
    -H "x-scrapengine-format: markdown" \
    -H "x-scrapengine-location: us" \
    -H "x-scrapengine-include-headers: true" \
    https://example.com/product
  ```

  ```python Python theme={null}
  import requests

  proxies = {
      'http': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081',
      'https': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'
  }

  headers = {
      'x-scrapengine-render': 'true',
      'x-scrapengine-format': 'markdown',
      'x-scrapengine-location': 'us',
      'x-scrapengine-include-headers': 'true'
  }

  response = requests.get('https://example.com/product', proxies=proxies, headers=headers)
  print(response.text)
  ```

  ```javascript Node.js theme={null}
  const axios = require('axios');

  const response = await axios.get('https://example.com/product', {
    proxy: {
      host: 'gw.scrapengine.io',
      port: 8081,
      auth: {
        username: 'scrape',
        password: 'YOUR_API_KEY'
      }
    },
    headers: {
      'x-scrapengine-render': 'true',
      'x-scrapengine-format': 'markdown',
      'x-scrapengine-location': 'us',
      'x-scrapengine-include-headers': 'true'
    }
  });

  console.log(response.data);
  ```
</CodeGroup>

### POST Requests

Send POST requests through the proxy:

<CodeGroup>
  ```bash cURL theme={null}
  curl -x http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081 \
    -X POST \
    -H "Content-Type: application/json" \
    -H "x-scrapengine-render: true" \
    -d '{"query": "search term"}' \
    https://example.com/api/search
  ```

  ```python Python theme={null}
  import requests

  proxies = {
      'http': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081',
      'https': 'http://scrape:YOUR_API_KEY@gw.scrapengine.io:8081'
  }

  headers = {
      'Content-Type': 'application/json',
      'x-scrapengine-render': 'true'
  }

  data = {'query': 'search term'}

  response = requests.post('https://example.com/api/search',
                          proxies=proxies,
                          headers=headers,
                          json=data)
  print(response.text)
  ```
</CodeGroup>

## Response Headers

The proxy returns the following headers with each response:

| Header                | Description                                              |
| --------------------- | -------------------------------------------------------- |
| `x-trace-id`          | Unique identifier for request tracing and support        |
| `x-remaining-credits` | Number of API credits remaining (on successful requests) |

## Error Handling

### Authentication Errors

If authentication fails, you'll receive a `407 Proxy Authentication Required` response:

```
HTTP/1.1 407 Proxy Authentication Required
Proxy-Authenticate: Basic realm="Proxy Authentication Required"
```

**Solutions:**

* Verify your API key is correct
* Ensure the password field contains your API key
* Check that your API key has not expired

### Common HTTP Errors

| Status | Description                                            |
| ------ | ------------------------------------------------------ |
| `400`  | Bad Request - Invalid URL or parameters                |
| `401`  | Unauthorized - Invalid API key                         |
| `407`  | Proxy Authentication Required - Missing credentials    |
| `408`  | Request Timeout - Target site took too long to respond |
| `503`  | Service Unavailable - Temporary service issue          |

### Error Response Format

```json theme={null}
{
  "status": "error",
  "message": "Error description",
  "traceId": "abc123-def456"
}
```

## Proxy vs REST API

| Feature            | Proxy                           | REST API                           |
| ------------------ | ------------------------------- | ---------------------------------- |
| **Integration**    | Works with any HTTP client      | Requires specific API calls        |
| **Authentication** | HTTP Basic Auth                 | Bearer token                       |
| **URL**            | `gw.scrapengine.io:8081`        | `api.scrapengine.io/api/v1/scrape` |
| **LLM Extraction** | Not supported                   | Fully supported                    |
| **Configuration**  | Via HTTP headers                | Via JSON body                      |
| **Best for**       | Existing tools, simple scraping | Advanced features, AI extraction   |

<Tip>
  Use the **REST API** if you need AI-powered data extraction with schemas or prompts. Use the **Proxy** for simple scraping or when integrating with existing tools that support proxy configuration.
</Tip>

## Use Cases

* **Browser automation tools**: Configure Puppeteer, Playwright, or Selenium to use Scrapengine as a proxy
* **CLI tools**: Use with wget, curl, or other command-line HTTP clients
* **Existing applications**: Add scraping capabilities without code changes
* **Testing tools**: Route traffic through Scrapengine for web testing
* **Scripting**: Simple one-liner scraping in shell scripts

## Best Practices

<AccordionGroup>
  <Accordion title="Use HTTPS targets when possible">
    Always prefer HTTPS URLs for better security and reliability when scraping.
  </Accordion>

  <Accordion title="Enable rendering only when needed">
    JavaScript rendering (`x-scrapengine-render: true`) uses more resources. Only enable it for pages that require JavaScript to load content.
  </Accordion>

  <Accordion title="Handle timeouts gracefully">
    Set appropriate timeouts in your HTTP client. Scraping can take longer than typical API calls, especially with rendering enabled.
  </Accordion>

  <Accordion title="Monitor your credits">
    Check the `x-remaining-credits` header in responses to monitor your usage and avoid unexpected interruptions.
  </Accordion>
</AccordionGroup>