completed, cancelled, or failed. Then you can page through its records as JSON or download them as files. Result files are deleted 7 days after the batch finishes.
Check status
GET /v1/batch/{batch_id} returns the batch. Poll every 10 to 30 seconds while status is queued, running, or cancelling, or let a webhook tell you when it’s done. A finished batch (trimmed):
A crawl can finish below its
maxUrls when it runs out of pages in scope.
Read results
GET /v1/batch/{batch_id}/results pages through a finished batch’s records as JSON. It returns 409 BATCH_NOT_COMPLETED until the batch is final.
limitis 1–100 and defaults to 25. A page can close early to stay under about 8 MB, so keep requesting withcursorset tonext_cursorwhilehas_moreistrue. An unknown cursor returns400.- Records aren’t in submission order. Match them with
itemId;metacomes back too. - A successful record has
status: "ok",final_url,metadata,cache_metadata, andmarkdownorhtml. Markdown batches withincludeHTMLreturn both.ocr_pagesappears when OCR recovered PDF pages. - A failed record has
status: "error",error_code, andmessage:
Download files
For large batches, download the files inresults.files instead of paging. Each file is gzipped NDJSON, one record per line, with up to 50 records, so a 25,000-URL batch has about 500 files. File order carries no meaning.
Each link expires 24 hours after the call that returned it (results.expires_at). Retrieve the batch again for new links. The data.batch.results links in a batch.* webhook expire 24 hours after the event, and a replay resends those same links.
Result files are deleted 7 days after the batch finishes. After that, the links stop working and /results can’t return the records, so copy what you need within 7 days.
List batches
GET /v1/batch/list returns your batches, newest first. It returns 25 per page by default and up to 100 with limit. Filter with these parameters:
Page with
cursor set to next_cursor. A search with q or tags can return 503 SEARCH_UNAVAILABLE; retry it, or list without them.
Related
- Submit a batch:
itemId,meta, and page options. - Cancel, limits, and errors: delete old batches and handle errors.
- List batch results reference and Retrieve a batch reference