> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Submit Batch Jobs

> Submit up to 25,000 URLs or crawl a site asynchronously, track progress, and consume results as paginated JSON or gzipped NDJSON.

Context.dev's Batch API runs large scraping jobs asynchronously. Submit a fixed list of up to 25,000 URLs, or start a crawl from one page or a domain's sitemap. Each successful page costs 1 credit; credits reserved for pages that do not succeed are released when the batch settles.

Choose Markdown or HTML output when you submit the batch. You can poll for progress, receive a webhook when processing finishes, page through results as JSON, or download gzipped NDJSON result files.

<Prompt description="Integrate Context.dev's Batch API in your app" icon="sparkles" actions={["copy", "cursor"]}>
  I'm integrating Context.dev's Batch API into my app. Help me:

  1. Read the API key from `CONTEXT_DEV_API_KEY`. Never hardcode it. Authenticate every request with a bearer token.
  2. Submit `POST /batch/submit` with either a `scrape` input containing up to 25,000 URL objects, or a `crawl` input using a `start_url` or `sitemap` source. Select `markdown` or `html` in `input.data.type`.
  3. Send a unique `Idempotency-Key` header so a retry cannot create or charge a duplicate batch. Save the returned batch `id`.
  4. Poll `GET /batch/{batch_id}` until `status` is `completed`, `cancelled`, or `failed`. If I provide `webhookUrl`, save `webhook_secret` and verify notifications.
  5. Read finished records from `GET /batch/{batch_id}/results` using `has_more` and `next_cursor`. For large batches, download the gzipped NDJSON URLs in `results.files` instead.
  6. Handle cancellation with `POST /batch/{batch_id}/cancel`, and permanently remove a finished batch with `DELETE /batch/{batch_id}`.
  7. Handle input errors, concurrent batch limits, idempotency conflicts, unfinished results, rate limits, and transient server errors explicitly.

  Docs: [https://docs.context.dev/guides/scrape-websites-in-batches](https://docs.context.dev/guides/scrape-websites-in-batches)
</Prompt>

## Prerequisites

Create an API key at [context.dev/signup](https://context.dev/signup), copy it from the [dashboard](https://context.dev/dashboard), and export it:

```bash theme={null}
export CONTEXT_DEV_API_KEY="ctxt_secret_..."
```

The examples below call the HTTPS API directly with `curl`.

## Choose how pages enter the batch

| Mode            | Input                                     | Behavior                                                                                |
| --------------- | ----------------------------------------- | --------------------------------------------------------------------------------------- |
| Fixed URL list  | `input.type: "scrape"`                    | Scrapes the URL objects in `input.data.urls`. Submit between 1 and 25,000.              |
| Start URL crawl | `input.type: "crawl"`, source `start_url` | Fetches the starting page, then follows eligible same-site links.                       |
| Sitemap batch   | `input.type: "crawl"`, source `sitemap`   | Reads a domain's sitemap and scrapes those URLs without following links from the pages. |

For every mode, set `input.data.type` to `markdown` or `html`. Markdown output supports link and image controls in addition to the common scraping options.

## Submit a fixed URL list

Each entry in `input.data.urls` requires a `url`. Add `itemId` or `meta` when you need to join result records back to your own data.

```bash cURL theme={null}
curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-import-2026-07-31-v1" \
  -d '{
    "input": {
      "type": "scrape",
      "data": {
        "type": "markdown",
        "urls": [
          {
            "url": "https://example.com/docs/getting-started",
            "itemId": "docs-001",
            "meta": { "section": "getting-started" }
          },
          {
            "url": "https://example.com/docs/api",
            "itemId": "docs-002"
          }
        ],
        "options": {
          "useMainContentOnly": true,
          "includeLinks": true,
          "maxAgeMs": 86400000
        }
      }
    },
    "webhookUrl": "https://example.com/webhooks/context",
    "tags": ["docs", "migration"]
  }'
```

<Badge color="blue" icon="coins">1 credit per successfully scraped page</Badge>

The API reserves credits for accepted pages at submission and returns `202`. Duplicate or invalid URLs are not accepted or charged.

```json sample response expandable theme={null}
{
  "id": "batch_9f2c8a",
  "status": "queued",
  "mode": "scrape",
  "type": "markdown",
  "tags": ["docs", "migration"],
  "input": {
    "submitted": 2,
    "accepted": 2,
    "duplicates": 0,
    "invalid": 0
  },
  "progress": {
    "succeeded": 0,
    "failed": 0,
    "pending": 2
  },
  "credits": {
    "estimated": 2,
    "charged": 0
  },
  "timing": {
    "created_at": "2026-07-31T12:00:00.000Z",
    "started_at": null,
    "completed_at": null
  },
  "errors": [],
  "error": null,
  "results": null,
  "invalid_urls": [],
  "webhook_secret": "whsec_...",
  "key_metadata": {
    "credits_consumed": 0,
    "credits_remaining": 500
  }
}
```

<Info>
  Retries with the same `Idempotency-Key` and request body return the original batch. Reusing the key with a different body returns `409 IDEMPOTENCY_KEY_CONFLICT`. Keys can contain up to 200 characters.
</Info>

### Content options

Common options under `input.data.options` include:

| Option                                  | Description                                                                         |
| --------------------------------------- | ----------------------------------------------------------------------------------- |
| `useMainContentOnly`                    | Remove navigation, headers, footers, and sidebars.                                  |
| `includeSelectors` / `excludeSelectors` | Keep or remove matching CSS subtrees.                                               |
| `maxAgeMs`                              | Reuse a cached scrape younger than this many milliseconds. Set `0` to scrape fresh. |
| `waitForMs`                             | Wait up to 15 seconds after browser load when rendering is needed.                  |
| `settleAnimations`                      | Wait briefly for CSS transitions and animations to settle.                          |
| `country`                               | Select the browser/proxy country.                                                   |
| `pdf`                                   | Control PDF parsing, page range, and embedded-image OCR.                            |

Markdown batches also support `includeLinks`, `includeImages`, and `shortenBase64Images`. See [Submit a Batch](/api-reference/batches/submit) for the complete schema.

## Crawl from a starting URL

Use a `start_url` source to follow links from one page. The starting URL is always included, even when `regex` does not match it. If the URL has no scheme, Context.dev reads it as HTTPS.

```bash cURL theme={null}
curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-docs-crawl-v1" \
  -d '{
    "input": {
      "type": "crawl",
      "data": {
        "type": "markdown",
        "source": {
          "type": "start_url",
          "url": "https://example.com/docs",
          "controls": {
            "maxUrls": 500,
            "maxDepth": 3,
            "followSubdomains": false,
            "regex": "^https://example\\.com/docs/"
          }
        },
        "options": {
          "useMainContentOnly": true
        }
      }
    },
    "tags": ["docs-crawl"]
  }'
```

`controls` is optional. `maxUrls` defaults to 100, `followSubdomains` defaults to `false`, and an omitted `maxDepth` allows any depth until another control stops discovery.

## Scrape URLs from a sitemap

Use a `sitemap` source to scrape URLs listed by a domain. The API accepts a bare domain or a full URL and reduces it to the domain.

```bash cURL theme={null}
curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-sitemap-v1" \
  -d '{
    "input": {
      "type": "crawl",
      "data": {
        "type": "html",
        "source": {
          "type": "sitemap",
          "domain": "example.com",
          "controls": {
            "maxUrls": 500,
            "regex": "^https://example\\.com/docs/"
          }
        }
      }
    }
  }'
```

<Warning>
  A sitemap batch scrapes only URLs from the sitemap. It does not follow links from those pages, so sitemap controls support `maxUrls` and `regex`, but not `maxDepth` or `followSubdomains`.
</Warning>

## Poll for completion

Save the `id` returned by submission, then retrieve the batch:

```bash cURL theme={null}
export BATCH_ID="batch_9f2c8a"

curl https://api.context.dev/v1/batch/$BATCH_ID \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"
```

Poll every few seconds while the status is `queued`, `running`, or `cancelling`. Stop when it becomes `completed`, `cancelled`, or `failed`.

| Field                | Meaning                                                             |
| -------------------- | ------------------------------------------------------------------- |
| `progress.succeeded` | Pages scraped successfully.                                         |
| `progress.failed`    | Pages that could not be scraped.                                    |
| `progress.pending`   | Accepted pages not yet attempted. It is `0` when a batch completes. |
| `credits.estimated`  | Credits reserved for accepted pages or the crawl page limit.        |
| `credits.charged`    | Credits consumed by successful pages.                               |
| `errors`             | Page failures grouped by error code.                                |
| `error`              | Batch-level failure detail when the batch itself fails.             |

A crawl can complete below `maxUrls` when no more eligible pages are reachable. The retrieve response also repeats `invalid_urls` and `webhook_secret`, so a dropped submit response does not lose them.

## Read result records as JSON

Once the batch is terminal, request a page of results:

```bash cURL theme={null}
curl -G https://api.context.dev/v1/batch/$BATCH_ID/results \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100"
```

Successful records have `status: "ok"` plus `http_status`, `final_url`, metadata, and either `markdown` or `html`. Failed records have `status: "error"`, `error_code`, and `message`. Scrape results echo any submitted `itemId` and `meta`.

```json sample response expandable theme={null}
{
  "data": [
    {
      "url": "https://example.com/docs/getting-started",
      "itemId": "docs-001",
      "meta": { "section": "getting-started" },
      "status": "ok",
      "http_status": 200,
      "final_url": "https://example.com/docs/getting-started",
      "markdown": "# Getting started\n\n...",
      "metadata": { "title": "Getting started" }
    },
    {
      "url": "https://example.com/missing",
      "status": "error",
      "error_code": "NOT_FOUND",
      "message": "Page returned 404"
    }
  ],
  "has_more": true,
  "next_cursor": "...",
  "key_metadata": {
    "credits_consumed": 0,
    "credits_remaining": 498
  }
}
```

A page can close before `limit` to keep its payload under approximately 8 MB. Continue based on `has_more` and `next_cursor`, not the number of records:

```bash cURL theme={null}
curl -G https://api.context.dev/v1/batch/$BATCH_ID/results \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100" \
  --data-urlencode "cursor=$NEXT_CURSOR"
```

For large result sets, retrieve the batch and download the signed URLs in `results.files` instead. Each file is gzipped NDJSON. The response includes `results.expires_at`; retrieve the batch again to obtain fresh links after they expire.

## Find previous batches

List batches newest first and combine status, search, and tag filters:

```bash cURL theme={null}
curl -G https://api.context.dev/v1/batch/list \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "status=completed" \
  --data-urlencode "q=docs" \
  --data-urlencode "search_type=prefix" \
  --data-urlencode "tags=docs,migration" \
  --data-urlencode "limit=50"
```

`q` matches batch IDs, crawl sources, and tags. The `tags` filter matches batches containing any supplied tag. Continue with `next_cursor` when `has_more` is true.

## Cancel or delete a batch

Cancellation stops new pages from starting. Pages already in progress finish before the batch reaches `cancelled`, and unused reserved credits are released.

```bash cURL theme={null}
curl -X POST https://api.context.dev/v1/batch/$BATCH_ID/cancel \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"
```

After a batch reaches `completed`, `cancelled`, or `failed`, you can permanently delete its metadata and stored results:

```bash cURL theme={null}
curl -X DELETE https://api.context.dev/v1/batch/$BATCH_ID \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"
```

## Handle errors

| Status | Error                                | What to do                                                              |
| ------ | ------------------------------------ | ----------------------------------------------------------------------- |
| 400    | Invalid request or unusable input    | Check the discriminated input shape, URL count, and strict field names. |
| 401    | Authentication or balance error      | Check the API key and available credits.                                |
| 403    | `BATCH_LIMIT_EXCEEDED`               | Wait for an active batch to finish before submitting another.           |
| 409    | `IDEMPOTENCY_KEY_CONFLICT`           | Use the original body or choose a new idempotency key.                  |
| 409    | `BATCH_NOT_COMPLETED`                | Wait for a terminal status before reading results.                      |
| 409    | Batch cannot be cancelled or deleted | Retrieve it and act based on its current status.                        |
| 429    | `RATE_LIMITED`                       | Wait for the rate-limit window and retry with backoff.                  |
| 500    | Input could not be staged or queued  | Retry with the same idempotency key so a duplicate is not created.      |

## Next steps

<CardGroup cols={2}>
  <Card title="Submit a Batch" icon="play" href="/api-reference/batches/submit">
    Full request schema, examples, and responses.
  </Card>

  <Card title="List Batch Results" icon="list" href="/api-reference/batches/results">
    Paginated success and error records.
  </Card>

  <Card title="Scrape Websites" icon="globe" href="/guides/scrape-websites-to-markdown">
    Use synchronous scraping for one page at a time.
  </Card>

  <Card title="Best Practices" icon="list-check" href="/optimization/best-practices">
    Authentication, retries, caching, and error handling.
  </Card>
</CardGroup>