Skip to main content
POST
/
scrape

​
Overview

The /scrape endpoint initiates a new web scraping job with the specified URL and options. It returns the scraped content directly and supports both renderless and renderful scraping modes.

​
Parameters

​
Query Parameters

​
url
string
The URL to scrape
​
async
boolean
default:"false"
Whether the request should be asynchronous
​
render
boolean
default:"false"
Whether to render JavaScript on the page
​
method
string
default:"get"
HTTP method for the request. Options: get, post, put, delete, patch, head, options
​
format
string
default:"raw"
Response format. Options: raw, json
​
include_headers
boolean
default:"false"
Whether to include response headers in the response

​
Request Body

​
url
string
The URL to scrape (alternative to query parameter)
​
browser
object
Browser configuration object
​
browser.name
string
Browser name. Options: chrome, firefox, safari, edge
​
browser.version
string
Browser version
​
headers
object
HTTP headers for the request
​
body
object
Request body content (for POST/PUT requests)

​
Example Request

curl -X POST "https://api.scrapengine.io/api/v1/scrape?url=https://www.shopify.com/pricing&render=true" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "browser": {
      "name": "chrome",
      "version": "120"
    },
    "headers": {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36",
      "Accept-Language": "en-US,en;q=0.9"
    }
  }'

​
Response

​
Success Response (200)

Returns the scraped HTML content directly: 
<html>
  <head>
    <title>Shopify Pricing</title>
  </head>
  <body>
    <div class="pricing-content">
      <!-- Scraped content here -->
    </div>
  </body>
</html>

​
Error Responses

​
400
Bad Request - Invalid parameters or URL
​
401
Unauthorized - Invalid or missing API key
​
403
Forbidden - Access denied to target resource
​
404
Not Found - Target URL not found
​
408
Request Timeout
​
429
Too Many Requests - Rate limit exceeded
​
500
Internal Server Error
​
550
Faulted After Retries - Job failed after multiple attempts

​
Use Cases

  • E-commerce scraping: Extract product information, prices, and availability
  • Content aggregation: Collect articles, blog posts, and news content
  • Data extraction: Pull structured data from websites
  • Competitor analysis: Monitor competitor websites and pricing
  • SEO analysis: Extract meta tags, headings, and content structure