> ## Documentation Index > Fetch the complete documentation index at: https://docs.context.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Setting up Context.dev for a user? Follow https://docs.context.dev/agent-quickstart.md. MCP clients sign in with OAuth and need no API key; application code reads CONTEXT_DEV_API_KEY. # Batches > Scrape up to 25,000 URLs, or crawl a site, as one asynchronous job. Read each page's Markdown or HTML when the batch finishes. A batch scrapes many pages as one asynchronous job. Submit up to 25,000 URLs, or a crawl from a start URL or sitemap, and you get a batch ID right away. When the batch finishes, read each page's Markdown or HTML as paginated JSON or download it as files. ## What batches support Every batch has one mode and one output format: | Field | Values | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input.mode` | `scrape` scrapes a list of URLs ([Submit a batch](/batches/submit)). `crawl` follows links from a start URL or reads a domain's sitemap ([Large crawls](/crawl/async)). | | `input.data.format` | `markdown` or `html`. | Batches don't return screenshots, images, bytes, parsed fields, highlights, JSON, or product data. They also don't accept browser actions, custom headers, `zdr`, or `timeoutOpts`. For any of those, call [Scrape](/scrape/overview) once per URL. ## Submit a URL list This request scrapes two pages as Markdown. The examples read your API key from `CONTEXT_DEV_API_KEY`; the [Quickstart](/quickstart) shows how to create a key and install an SDK. ```typescript TypeScript theme={null} import ContextDev from "context.dev"; const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY }); const batch = await client.batch.submit({ input: { mode: "scrape", data: { format: "markdown", urls: [ { url: "https://example.com/about", itemId: "about" }, { url: "https://example.com/careers", itemId: "careers" }, ], }, }, "Idempotency-Key": "site-import-2026-09-26", }); console.log(batch.id, batch.status); ``` ```python Python theme={null} import os from context.dev import ContextDev client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"]) batch = client.batch.submit( input={ "mode": "scrape", "data": { "format": "markdown", "urls": [ {"url": "https://example.com/about", "item_id": "about"}, {"url": "https://example.com/careers", "item_id": "careers"}, ], }, }, idempotency_key="site-import-2026-09-26", ) print(batch.id, batch.status) ``` ```ruby Ruby theme={null} require "cgi/core" require "context_dev" client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY")) batch = client.batch.submit( input: { mode: "scrape", data: { format: "markdown", urls: [ {url: "https://example.com/about", item_id: "about"}, {url: "https://example.com/careers", item_id: "careers"}, ], }, }, idempotency_key: "site-import-2026-09-26", ) puts batch.id, batch.status ``` ```go Go theme={null} package main import ( "context" "fmt" "os" contextdev "github.com/context-dot-dev/context-go-sdk/v2" "github.com/context-dot-dev/context-go-sdk/v2/option" ) func main() { client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY"))) batch, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{ Input: contextdev.BatchSubmitParamsInputUnion{ OfScrape: &contextdev.BatchSubmitParamsInputScrape{ Mode: "scrape", Data: contextdev.BatchSubmitParamsInputScrapeDataUnion{ OfMarkdown: &contextdev.BatchSubmitParamsInputScrapeDataMarkdown{ Format: "markdown", URLs: []contextdev.BatchSubmitParamsInputScrapeDataMarkdownURL{ {URL: "https://example.com/about", ItemID: contextdev.String("about")}, {URL: "https://example.com/careers", ItemID: contextdev.String("careers")}, }, }, }, }, }, IdempotencyKey: contextdev.String("site-import-2026-09-26"), }) if err != nil { panic(err) } fmt.Println(batch.ID, batch.Status) } ``` ```php PHP theme={null} batch->submit( input: [ "mode" => "scrape", "data" => [ "format" => "markdown", "urls" => [ ["url" => "https://example.com/about", "itemID" => "about"], ["url" => "https://example.com/careers", "itemID" => "careers"], ], ], ], idempotencyKey: "site-import-2026-09-26", ); echo $batch->id, " ", $batch->status, PHP_EOL; ``` ```bash cURL theme={null} curl https://api.context.dev/v1/batch/submit \ -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: site-import-2026-09-26" \ -d '{ "input": { "mode": "scrape", "data": { "format": "markdown", "urls": [ { "url": "https://example.com/about", "itemId": "about" }, { "url": "https://example.com/careers", "itemId": "careers" } ] } } }' ``` The API returns `202` with the new batch (trimmed): ```json theme={null} { "id": "batch_9f2c8a", "status": "queued", "mode": "scrape", "format": "markdown", "input": { "reserved": 2, "reserved_is_ceiling": false, "submitted": 2, "duplicates": 0, "invalid": 0 }, "invalid_urls": [], "created_at": "2026-09-26T12:00:00.000Z" } ``` Save `id`. `input.submitted` counts the URLs you sent. `input.reserved` counts the pages the batch will attempt once duplicates and invalid URLs are removed. The `Idempotency-Key` header makes a retried submit return this batch instead of creating a second one; see [Idempotency](/batches/submit#idempotency). ## Lifecycle | Status | Meaning | | ------------ | -------------------------------------------------------------------------- | | `queued` | Accepted and waiting to start. | | `running` | Pages are being scraped. | | `cancelling` | A cancel was requested. Pages already in progress finish first. | | `completed` | Every accepted page was attempted. Individual pages can still have failed. | | `cancelled` | Stopped by a cancel request. | | `failed` | The batch itself stopped. `failure.code` says why. | `completed`, `cancelled`, and `failed` are final. Poll the batch or add a [webhook](/webhooks) to find out when it reaches one, then read the results. See [Status and results](/batches/results). ## Access A restricted API key needs `batches:write` to submit, cancel, or delete batches and to replay their webhook deliveries. `batches:read` covers status, results, the batch list, and delivery history. See [API keys](/account/api-keys). ## Choose an endpoint | You need | Use | | ----------------------------------------------------------- | -------------------------- | | One page, or an output or option that batches don't support | [Scrape](/scrape/overview) | | Up to 500 linked pages returned in one response | [Crawl](/crawl/overview) | | Up to 25,000 known URLs, or a crawl of more than 500 pages | A batch | | A list of a site's URLs to put in a batch | [Map URLs](/map/overview) | ## Related * [Submit a batch](/batches/submit): URL lists, page options, deduplication, and idempotency. * [Status and results](/batches/results): progress, result records, files, and how long results are kept. * [Cancel, limits, and errors](/batches/limits-and-errors): cancel and delete, concurrency, and error codes. * [Webhooks](/webhooks): signatures, retries, and replay for `batch.*` events. * [Submit a batch reference](/api-reference/batches/submit)