Skip to main content
Submit 1–25,000 URL records to POST /batch/submit. Use the Batches overview for runnable SDK examples, or Large crawls for discovery from a site.

URL records and format

itemId and meta are returned with each page. Duplicate URLs are deduplicated; rejected URLs appear in invalid_urls (up to 100 entries). Keep the request body within 8 MB and use public HTTP(S) URLs. Choose one format: markdown or html. includeHTML adds HTML alongside Markdown. Other Scrape outputs, browser actions, custom headers, ZDR, and request deadlines are not supported by batches.

Per-page options

Use up to 20 request tags. Configure webhook: {"url":"https://app.example.com/hooks/context","retry":{}} for signed completion events; do not send webhookUrl at the same time.

Idempotency

Send an Idempotency-Key header, up to 200 characters, unique within your organization. Retrying the same key and body returns the original batch; a different body returns 409. Keep the key and payload together when retrying an uncertain submission.

Recover selected pages

Read result records, select failed URLs, and submit them as a new batch with a new idempotency key. Use maxAgeMs: 0 for a fresh price list, or remove it when recent cached pages are acceptable. Check limits and errors before increasing concurrency.