HTTP Proxy

Overview

The Scrapengine Proxy allows you to use Scrapengine as a standard HTTP/HTTPS proxy server. This is ideal for integrating with existing tools, libraries, or applications that support proxy configuration. Simply point your HTTP client to our proxy endpoint and authenticate using your API key.

The proxy interface provides the same scraping capabilities as the REST API but through a standard proxy protocol that works with any HTTP client.

Proxy Endpoint

gw.scrapengine.io:8081

Authentication

Authentication is done via HTTP Basic Auth where:

Username: Any value (e.g., scrape)
Password: Your Scrapengine API key

Quick Start

curl -x http://scrape:[email protected]:8081 \
  https://example.com

const axios = require('axios');

const response = await axios.get('https://example.com', {
  proxy: {
    host: 'gw.scrapengine.io',
    port: 8081,
    auth: {
      username: 'scrape',
      password: 'YOUR_API_KEY'
    }
  }
});

console.log(response.data);

import requests

proxies = {
    'http': 'http://scrape:[email protected]:8081',
    'https': 'http://scrape:[email protected]:8081'
}

response = requests.get('https://example.com', proxies=proxies)
print(response.text)

import httpx

proxy = 'http://scrape:[email protected]:8081'

with httpx.Client(proxy=proxy) as client:
    response = client.get('https://example.com')
    print(response.text)

const fetch = require('node-fetch');
const { HttpsProxyAgent } = require('https-proxy-agent');

const agent = new HttpsProxyAgent('http://scrape:[email protected]:8081');

const response = await fetch('https://example.com', { agent });
console.log(await response.text());

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
)

func main() {
    proxyURL, _ := url.Parse("http://scrape:[email protected]:8081")

    client := &http.Client{
        Transport: &http.Transport{
            Proxy: http.ProxyURL(proxyURL),
        },
    }

    resp, _ := client.Get("https://example.com")
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    fmt.Println(string(body))
}

Configuration Headers

Control scraping behavior by passing custom headers with your request. These headers are parsed by the proxy and configure how the scraping job is executed.

Header	Type	Default	Description
`x-scrapengine-render`	boolean	`false`	Enable JavaScript rendering for dynamic content
`x-scrapengine-async`	boolean	`false`	Return immediately with job ID for async processing
`x-scrapengine-location`	string	`us`	Proxy country code (e.g., `us`, `uk`, `de`)
`x-scrapengine-format`	string	`raw`	Response format: `raw`, `json`, or `markdown`
`x-scrapengine-include-headers`	boolean	`false`	Include response headers in the response
`x-scrapengine-verbose`	boolean	`false`	Return detailed response headers

Examples with Headers

Enable JavaScript Rendering

Scrape pages that require JavaScript to load content:

curl -x http://scrape:[email protected]:8081 \
  -H "x-scrapengine-render: true" \
  https://example.com/dynamic-page

import requests

proxies = {
    'http': 'http://scrape:[email protected]:8081',
    'https': 'http://scrape:[email protected]:8081'
}

headers = {
    'x-scrapengine-render': 'true'
}

response = requests.get('https://example.com/dynamic-page', proxies=proxies, headers=headers)
print(response.text)

const axios = require('axios');

const response = await axios.get('https://example.com/dynamic-page', {
  proxy: {
    host: 'gw.scrapengine.io',
    port: 8081,
    auth: {
      username: 'scrape',
      password: 'YOUR_API_KEY'
    }
  },
  headers: {
    'x-scrapengine-render': 'true'
  }
});

console.log(response.data);

Get Markdown Output

Convert the scraped content to clean Markdown:

curl -x http://scrape:[email protected]:8081 \
  -H "x-scrapengine-format: markdown" \
  https://example.com/article

import requests

proxies = {
    'http': 'http://scrape:[email protected]:8081',
    'https': 'http://scrape:[email protected]:8081'
}

headers = {
    'x-scrapengine-format': 'markdown'
}

response = requests.get('https://example.com/article', proxies=proxies, headers=headers)
print(response.text)

Geo-Targeted Scraping

Scrape from a specific geographic location:

curl -x http://scrape:[email protected]:8081 \
  -H "x-scrapengine-location: uk" \
  -H "x-scrapengine-render: true" \
  https://example.com/local-prices

import requests

proxies = {
    'http': 'http://scrape:[email protected]:8081',
    'https': 'http://scrape:[email protected]:8081'
}

headers = {
    'x-scrapengine-location': 'uk',
    'x-scrapengine-render': 'true'
}

response = requests.get('https://example.com/local-prices', proxies=proxies, headers=headers)
print(response.text)

Multiple Options Combined

Combine multiple options for advanced scraping:

curl -x http://scrape:[email protected]:8081 \
  -H "x-scrapengine-render: true" \
  -H "x-scrapengine-format: markdown" \
  -H "x-scrapengine-location: us" \
  -H "x-scrapengine-include-headers: true" \
  https://example.com/product

import requests

proxies = {
    'http': 'http://scrape:[email protected]:8081',
    'https': 'http://scrape:[email protected]:8081'
}

headers = {
    'x-scrapengine-render': 'true',
    'x-scrapengine-format': 'markdown',
    'x-scrapengine-location': 'us',
    'x-scrapengine-include-headers': 'true'
}

response = requests.get('https://example.com/product', proxies=proxies, headers=headers)
print(response.text)

const axios = require('axios');

const response = await axios.get('https://example.com/product', {
  proxy: {
    host: 'gw.scrapengine.io',
    port: 8081,
    auth: {
      username: 'scrape',
      password: 'YOUR_API_KEY'
    }
  },
  headers: {
    'x-scrapengine-render': 'true',
    'x-scrapengine-format': 'markdown',
    'x-scrapengine-location': 'us',
    'x-scrapengine-include-headers': 'true'
  }
});

console.log(response.data);

POST Requests

Send POST requests through the proxy:

curl -x http://scrape:[email protected]:8081 \
  -X POST \
  -H "Content-Type: application/json" \
  -H "x-scrapengine-render: true" \
  -d '{"query": "search term"}' \
  https://example.com/api/search

import requests

proxies = {
    'http': 'http://scrape:[email protected]:8081',
    'https': 'http://scrape:[email protected]:8081'
}

headers = {
    'Content-Type': 'application/json',
    'x-scrapengine-render': 'true'
}

data = {'query': 'search term'}

response = requests.post('https://example.com/api/search',
                        proxies=proxies,
                        headers=headers,
                        json=data)
print(response.text)

Response Headers

The proxy returns the following headers with each response:

Header	Description
`x-trace-id`	Unique identifier for request tracing and support
`x-remaining-credits`	Number of API credits remaining (on successful requests)

Error Handling

Authentication Errors

If authentication fails, you’ll receive a 407 Proxy Authentication Required response:

HTTP/1.1 407 Proxy Authentication Required
Proxy-Authenticate: Basic realm="Proxy Authentication Required"

Solutions:

Verify your API key is correct
Ensure the password field contains your API key
Check that your API key has not expired

Common HTTP Errors

Status	Description
`400`	Bad Request - Invalid URL or parameters
`401`	Unauthorized - Invalid API key
`407`	Proxy Authentication Required - Missing credentials
`408`	Request Timeout - Target site took too long to respond
`503`	Service Unavailable - Temporary service issue

Error Response Format

{
  "status": "error",
  "message": "Error description",
  "traceId": "abc123-def456"
}

Proxy vs REST API

Feature	Proxy	REST API
Integration	Works with any HTTP client	Requires specific API calls
Authentication	HTTP Basic Auth	Bearer token
URL	`gw.scrapengine.io:8081`	`api.scrapengine.io/api/v1/scrape`
LLM Extraction	Not supported	Fully supported
Configuration	Via HTTP headers	Via JSON body
Best for	Existing tools, simple scraping	Advanced features, AI extraction

Use the REST API if you need AI-powered data extraction with schemas or prompts. Use the Proxy for simple scraping or when integrating with existing tools that support proxy configuration.

Use Cases

Browser automation tools: Configure Puppeteer, Playwright, or Selenium to use Scrapengine as a proxy
CLI tools: Use with wget, curl, or other command-line HTTP clients
Existing applications: Add scraping capabilities without code changes
Testing tools: Route traffic through Scrapengine for web testing
Scripting: Simple one-liner scraping in shell scripts

Best Practices

Use HTTPS targets when possible

Always prefer HTTPS URLs for better security and reliability when scraping.

Enable rendering only when needed

JavaScript rendering (x-scrapengine-render: true) uses more resources. Only enable it for pages that require JavaScript to load content.

Handle timeouts gracefully

Set appropriate timeouts in your HTTP client. Scraping can take longer than typical API calls, especially with rendering enabled.

Monitor your credits

Check the x-remaining-credits header in responses to monitor your usage and avoid unexpected interruptions.

API documentation

Web Scraping

Browser Sessions

Sessions

Tools

Tasks

AI

Recording

Extensions

Session Proxy

Clipboard

Filesystem

Session Files

Credentials & Vault

Search Engines

E-Commerce

Overview

Proxy Endpoint

Authentication

Quick Start

Configuration Headers

Examples with Headers

Enable JavaScript Rendering

Get Markdown Output

Geo-Targeted Scraping

Multiple Options Combined

POST Requests

Response Headers

Error Handling

Authentication Errors

Common HTTP Errors

Error Response Format

Proxy vs REST API

Use Cases

Best Practices

​Overview

​Proxy Endpoint

​Authentication

​Quick Start

​Configuration Headers

​Examples with Headers

​Enable JavaScript Rendering

​Get Markdown Output

​Geo-Targeted Scraping

​Multiple Options Combined

​POST Requests

​Response Headers

​Error Handling

​Authentication Errors

​Common HTTP Errors

​Error Response Format

​Proxy vs REST API

​Use Cases

​Best Practices

Overview

Proxy Endpoint

Authentication

Quick Start

Configuration Headers

Examples with Headers

Enable JavaScript Rendering

Get Markdown Output

Geo-Targeted Scraping

Multiple Options Combined

POST Requests

Response Headers

Error Handling

Authentication Errors

Common HTTP Errors

Error Response Format

Proxy vs REST API

Use Cases

Best Practices