Submit Batch Jobs

Context.dev’s Batch API runs large scraping jobs asynchronously. Submit a fixed list of up to 25,000 URLs, or start a crawl from one page or a domain’s sitemap. Each successful page costs 1 credit; credits reserved for pages that do not succeed are released when the batch settles. Choose Markdown or HTML output when you submit the batch. You can poll for progress, receive a webhook when processing finishes, page through results as JSON, or download gzipped NDJSON result files.

Integrate Context.dev's Batch API in your app

Open in Cursor

Prerequisites

Create an API key at context.dev/signup, copy it from the dashboard, and export it:

export CONTEXT_DEV_API_KEY="ctxt_secret_..."

The examples below call the HTTPS API directly with curl.

Choose how pages enter the batch

Mode	Input	Behavior
Fixed URL list	`input.type: "scrape"`	Scrapes the URL objects in `input.data.urls`. Submit between 1 and 25,000.
Start URL crawl	`input.type: "crawl"`, source `start_url`	Fetches the starting page, then follows eligible same-site links.
Sitemap batch	`input.type: "crawl"`, source `sitemap`	Reads a domain’s sitemap and scrapes those URLs without following links from the pages.

For every mode, set input.data.type to markdown or html. Markdown output supports link and image controls in addition to the common scraping options.

Submit a fixed URL list

Each entry in input.data.urls requires a url. Add itemId or meta when you need to join result records back to your own data.

cURL

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-import-2026-07-31-v1" \
  -d '{
    "input": {
      "type": "scrape",
      "data": {
        "type": "markdown",
        "urls": [
          {
            "url": "https://example.com/docs/getting-started",
            "itemId": "docs-001",
            "meta": { "section": "getting-started" }
          },
          {
            "url": "https://example.com/docs/api",
            "itemId": "docs-002"
          }
        ],
        "options": {
          "useMainContentOnly": true,
          "includeLinks": true,
          "maxAgeMs": 86400000
        }
      }
    },
    "webhookUrl": "https://example.com/webhooks/context",
    "tags": ["docs", "migration"]
  }'

1 credit per successfully scraped page The API reserves credits for accepted pages at submission and returns 202. Duplicate or invalid URLs are not accepted or charged.

sample response

{
  "id": "batch_9f2c8a",
  "status": "queued",
  "mode": "scrape",
  "type": "markdown",
  "tags": ["docs", "migration"],
  "input": {
    "submitted": 2,
    "accepted": 2,
    "duplicates": 0,
    "invalid": 0
  },
  "progress": {
    "succeeded": 0,
    "failed": 0,
    "pending": 2
  },
  "credits": {
    "estimated": 2,
    "charged": 0
  },
  "timing": {
    "created_at": "2026-07-31T12:00:00.000Z",
    "started_at": null,
    "completed_at": null
  },
  "errors": [],
  "error": null,
  "results": null,
  "invalid_urls": [],
  "webhook_secret": "whsec_...",
  "key_metadata": {
    "credits_consumed": 0,
    "credits_remaining": 500
  }
}

Retries with the same Idempotency-Key and request body return the original batch. Reusing the key with a different body returns 409 IDEMPOTENCY_KEY_CONFLICT. Keys can contain up to 200 characters.

Content options

Common options under input.data.options include:

Option	Description
`useMainContentOnly`	Remove navigation, headers, footers, and sidebars.
`includeSelectors` / `excludeSelectors`	Keep or remove matching CSS subtrees.
`maxAgeMs`	Reuse a cached scrape younger than this many milliseconds. Set `0` to scrape fresh.
`waitForMs`	Wait up to 15 seconds after browser load when rendering is needed.
`settleAnimations`	Wait briefly for CSS transitions and animations to settle.
`country`	Select the browser/proxy country.
`pdf`	Control PDF parsing, page range, and embedded-image OCR.

Markdown batches also support includeLinks, includeImages, and shortenBase64Images. See Submit a Batch for the complete schema.

Crawl from a starting URL

Use a start_url source to follow links from one page. The starting URL is always included, even when regex does not match it. If the URL has no scheme, Context.dev reads it as HTTPS.

cURL

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-docs-crawl-v1" \
  -d '{
    "input": {
      "type": "crawl",
      "data": {
        "type": "markdown",
        "source": {
          "type": "start_url",
          "url": "https://example.com/docs",
          "controls": {
            "maxUrls": 500,
            "maxDepth": 3,
            "followSubdomains": false,
            "regex": "^https://example\\.com/docs/"
          }
        },
        "options": {
          "useMainContentOnly": true
        }
      }
    },
    "tags": ["docs-crawl"]
  }'

controls is optional. maxUrls defaults to 100, followSubdomains defaults to false, and an omitted maxDepth allows any depth until another control stops discovery.

Scrape URLs from a sitemap

Use a sitemap source to scrape URLs listed by a domain. The API accepts a bare domain or a full URL and reduces it to the domain.

cURL

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-sitemap-v1" \
  -d '{
    "input": {
      "type": "crawl",
      "data": {
        "type": "html",
        "source": {
          "type": "sitemap",
          "domain": "example.com",
          "controls": {
            "maxUrls": 500,
            "regex": "^https://example\\.com/docs/"
          }
        }
      }
    }
  }'

A sitemap batch scrapes only URLs from the sitemap. It does not follow links from those pages, so sitemap controls support maxUrls and regex, but not maxDepth or followSubdomains.

Poll for completion

Save the id returned by submission, then retrieve the batch:

cURL

export BATCH_ID="batch_9f2c8a"

curl https://api.context.dev/v1/batch/$BATCH_ID \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

Poll every few seconds while the status is queued, running, or cancelling. Stop when it becomes completed, cancelled, or failed.

Field	Meaning
`progress.succeeded`	Pages scraped successfully.
`progress.failed`	Pages that could not be scraped.
`progress.pending`	Accepted pages not yet attempted. It is `0` when a batch completes.
`credits.estimated`	Credits reserved for accepted pages or the crawl page limit.
`credits.charged`	Credits consumed by successful pages.
`errors`	Page failures grouped by error code.
`error`	Batch-level failure detail when the batch itself fails.

A crawl can complete below maxUrls when no more eligible pages are reachable. The retrieve response also repeats invalid_urls and webhook_secret, so a dropped submit response does not lose them.

Read result records as JSON

Once the batch is terminal, request a page of results:

cURL

curl -G https://api.context.dev/v1/batch/$BATCH_ID/results \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100"

Successful records have status: "ok" plus http_status, final_url, metadata, and either markdown or html. Failed records have status: "error", error_code, and message. Scrape results echo any submitted itemId and meta.

sample response

{
  "data": [
    {
      "url": "https://example.com/docs/getting-started",
      "itemId": "docs-001",
      "meta": { "section": "getting-started" },
      "status": "ok",
      "http_status": 200,
      "final_url": "https://example.com/docs/getting-started",
      "markdown": "# Getting started\n\n...",
      "metadata": { "title": "Getting started" }
    },
    {
      "url": "https://example.com/missing",
      "status": "error",
      "error_code": "NOT_FOUND",
      "message": "Page returned 404"
    }
  ],
  "has_more": true,
  "next_cursor": "...",
  "key_metadata": {
    "credits_consumed": 0,
    "credits_remaining": 498
  }
}

A page can close before limit to keep its payload under approximately 8 MB. Continue based on has_more and next_cursor, not the number of records:

cURL

curl -G https://api.context.dev/v1/batch/$BATCH_ID/results \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100" \
  --data-urlencode "cursor=$NEXT_CURSOR"

For large result sets, retrieve the batch and download the signed URLs in results.files instead. Each file is gzipped NDJSON. The response includes results.expires_at; retrieve the batch again to obtain fresh links after they expire.

Find previous batches

List batches newest first and combine status, search, and tag filters:

cURL

curl -G https://api.context.dev/v1/batch/list \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "status=completed" \
  --data-urlencode "q=docs" \
  --data-urlencode "search_type=prefix" \
  --data-urlencode "tags=docs,migration" \
  --data-urlencode "limit=50"

q matches batch IDs, crawl sources, and tags. The tags filter matches batches containing any supplied tag. Continue with next_cursor when has_more is true.

Cancel or delete a batch

Cancellation stops new pages from starting. Pages already in progress finish before the batch reaches cancelled, and unused reserved credits are released.

cURL

curl -X POST https://api.context.dev/v1/batch/$BATCH_ID/cancel \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

After a batch reaches completed, cancelled, or failed, you can permanently delete its metadata and stored results:

cURL

curl -X DELETE https://api.context.dev/v1/batch/$BATCH_ID \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

Handle errors

Status	Error	What to do
400	Invalid request or unusable input	Check the discriminated input shape, URL count, and strict field names.
401	Authentication or balance error	Check the API key and available credits.
403	`BATCH_LIMIT_EXCEEDED`	Wait for an active batch to finish before submitting another.
409	`IDEMPOTENCY_KEY_CONFLICT`	Use the original body or choose a new idempotency key.
409	`BATCH_NOT_COMPLETED`	Wait for a terminal status before reading results.
409	Batch cannot be cancelled or deleted	Retrieve it and act based on its current status.
429	`RATE_LIMITED`	Wait for the rate-limit window and retry with backoff.
500	Input could not be staged or queued	Retry with the same idempotency key so a duplicate is not created.

Next steps

Submit a Batch

Full request schema, examples, and responses.

List Batch Results

Paginated success and error records.

Scrape Websites

Use synchronous scraping for one page at a time.

Best Practices

Authentication, retries, caching, and error handling.

Get Started

Give it to your Agent

What can Context.dev do

Optimizations

No-code Integrations

Prerequisites

Choose how pages enter the batch

Submit a fixed URL list

Content options

Crawl from a starting URL

Scrape URLs from a sitemap

Poll for completion

Read result records as JSON

Find previous batches

Cancel or delete a batch

Handle errors

Next steps

Submit a Batch

List Batch Results

Scrape Websites

Best Practices

​Prerequisites

​Choose how pages enter the batch

​Submit a fixed URL list

​Content options

​Crawl from a starting URL

​Scrape URLs from a sitemap

​Poll for completion

​Read result records as JSON

​Find previous batches

​Cancel or delete a batch

​Handle errors

​Next steps

Submit a Batch

List Batch Results

Scrape Websites

Best Practices

Prerequisites

Choose how pages enter the batch

Submit a fixed URL list

Content options

Crawl from a starting URL

Scrape URLs from a sitemap

Poll for completion

Read result records as JSON

Find previous batches

Cancel or delete a batch

Handle errors

Next steps