Skip to main content
A batch is done when its status is completed, cancelled, or failed. Then you can page through its records as JSON or download them as files. Result files are deleted 7 days after the batch finishes.

Check status

GET /v1/batch/{batch_id} returns the batch. Poll every 10 to 30 seconds while status is queued, running, or cancelling, or let a webhook tell you when it’s done. A finished batch (trimmed):
A crawl can finish below its maxUrls when it runs out of pages in scope.

Read results

GET /v1/batch/{batch_id}/results pages through a finished batch’s records as JSON. It returns 409 BATCH_NOT_COMPLETED until the batch is final.
  • limit is 1–100 and defaults to 25. A page can close early to stay under about 8 MB, so keep requesting with cursor set to next_cursor while has_more is true. An unknown cursor returns 400.
  • Records aren’t in submission order. Match them with itemId; meta comes back too.
  • A successful record has status: "ok", final_url, metadata, cache_metadata, and markdown or html. Markdown batches with includeHTML return both. ocr_pages appears when OCR recovered PDF pages.
  • A failed record has status: "error", error_code, and message:

Download files

For large batches, download the files in results.files instead of paging. Each file is gzipped NDJSON, one record per line, with up to 50 records, so a 25,000-URL batch has about 500 files. File order carries no meaning. Each link expires 24 hours after the call that returned it (results.expires_at). Retrieve the batch again for new links. The data.batch.results links in a batch.* webhook expire 24 hours after the event, and a replay resends those same links. Result files are deleted 7 days after the batch finishes. After that, the links stop working and /results can’t return the records, so copy what you need within 7 days.

List batches

GET /v1/batch/list returns your batches, newest first. It returns 25 per page by default and up to 100 with limit. Filter with these parameters: Page with cursor set to next_cursor. A search with q or tags can return 503 SEARCH_UNAVAILABLE; retry it, or list without them.