Scrape in batches

Use the Batch API to scrape up to 25,000 URLs asynchronously, or discover pages from a starting URL or sitemap. Choose Markdown or HTML, then poll for completion and download the results.

Already have the URLs? Use this batch scraping guide. To discover pages from a website, follow Crawl Async (25K pages). For a small linked section returned in one response, use Crawl Sync.

Each successful page costs 1 credit, plus any PDF OCR charges. Credits reserved for pages that do not succeed are released when the batch settles. Management calls consume no credits.

Prerequisites

Create an API key at context.dev/signup, copy it from the dashboard, and export it:

export CONTEXT_DEV_API_KEY="ctxt_secret_..."

Choose cURL or install an SDK for your language. Each example creates its own client. A restricted key needs batches:write to submit or manage jobs; batches:read can read status and results. See API key permissions.

Choose an input

Mode	Input	Behavior
Fixed URL list	`input.mode: "scrape"`	Scrapes the URL objects in `input.data.urls`. Submit between 1 and 25,000.
Start URL crawl	`input.mode: "crawl"`, source `start_url`	Fetches the starting page, then follows eligible same-site links.
Sitemap batch	`input.mode: "crawl"`, source `sitemap`	Reads a domain’s sitemap and scrapes those URLs without following links from the pages.

For every mode, set input.data.format to markdown or html. Markdown output supports link and image controls in addition to the common scraping options.

Submit a fixed URL list

Each entry requires a url. Add itemId or meta to match results to your records. Choose a new Idempotency-Key for each new job and reuse it when retrying that job.

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-import-2026-07-31-v1" \
  -d '{
    "input": {
      "mode": "scrape",
      "data": {
        "format": "markdown",
        "urls": [
          {
            "url": "https://docs.context.dev/introduction",
            "itemId": "docs-001",
            "meta": { "section": "getting-started" }
          },
          {
            "url": "https://docs.context.dev/quickstart",
            "itemId": "docs-002"
          }
        ],
        "options": {
          "useMainContentOnly": true,
          "includeLinks": true,
          "maxAgeMs": 86400000
        }
      }
    },
    "tags": ["docs", "migration"]
  }'

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.submit({
  input: {
    mode: "scrape",
    data: {
      format: "markdown",
      urls: [
        {
          url: "https://docs.context.dev/introduction",
          itemId: "docs-001",
          meta: {
            section: "getting-started",
          },
        },
        {
          url: "https://docs.context.dev/quickstart",
          itemId: "docs-002",
        },
      ],
      options: {
        useMainContentOnly: true,
        includeLinks: true,
        maxAgeMs: 86400000,
      },
    },
  },
  tags: ["docs", "migration"],
  "Idempotency-Key": "docs-import-2026-07-31-v1",
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.submit(
    input={
        "mode": "scrape",
        "data": {
            "format": "markdown",
            "urls": [
                {
                    "url": "https://docs.context.dev/introduction",
                    "item_id": "docs-001",
                    "meta": {
                        "section": "getting-started",
                    },
                },
                {
                    "url": "https://docs.context.dev/quickstart",
                    "item_id": "docs-002",
                },
            ],
            "options": {
                "use_main_content_only": True,
                "include_links": True,
                "max_age_ms": 86400000,
            },
        },
    },
    tags=["docs", "migration"],
    idempotency_key="docs-import-2026-07-31-v1",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.submit(
  input: {
    mode: "scrape",
    data: {
      format: "markdown",
      urls: [
        {
          url: "https://docs.context.dev/introduction",
          item_id: "docs-001",
          meta: {
            section: "getting-started",
          },
        },
        {
          url: "https://docs.context.dev/quickstart",
          item_id: "docs-002",
        },
      ],
      options: {
        use_main_content_only: true,
        include_links: true,
        max_age_ms: 86400000,
      },
    },
  },
  tags: ["docs", "migration"],
  idempotency_key: "docs-import-2026-07-31-v1",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
		Input: contextdev.BatchSubmitParamsInputUnion{
			OfScrape: &contextdev.BatchSubmitParamsInputScrape{
				Mode: "scrape",
				Data: contextdev.BatchSubmitParamsInputScrapeDataUnion{
					OfMarkdown: &contextdev.BatchSubmitParamsInputScrapeDataMarkdown{
						Format: "markdown",
						URLs: []contextdev.BatchSubmitParamsInputScrapeDataMarkdownURL{
							contextdev.BatchSubmitParamsInputScrapeDataMarkdownURL{
								URL:    "https://docs.context.dev/introduction",
								ItemID: contextdev.String("docs-001"),
								Meta: map[string]any{
									"section": "getting-started",
								},
							},
							contextdev.BatchSubmitParamsInputScrapeDataMarkdownURL{
								URL:    "https://docs.context.dev/quickstart",
								ItemID: contextdev.String("docs-002"),
							},
						},
						Options: contextdev.BatchSubmitParamsInputScrapeDataMarkdownOptions{
							UseMainContentOnly: contextdev.Bool(true),
							IncludeLinks:       contextdev.Bool(true),
							MaxAgeMs:           contextdev.Int(86400000),
						},
					},
				},
			},
		},
		Tags:           []string{"docs", "migration"},
		IdempotencyKey: contextdev.String("docs-import-2026-07-31-v1"),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->submit(
    input: [
        "mode" => "scrape",
        "data" => [
            "format" => "markdown",
            "urls" => [
                [
                    "url" => "https://docs.context.dev/introduction",
                    "itemID" => "docs-001",
                    "meta" => [
                        "section" => "getting-started",
                    ],
                ],
                [
                    "url" => "https://docs.context.dev/quickstart",
                    "itemID" => "docs-002",
                ],
            ],
            "options" => [
                "useMainContentOnly" => true,
                "includeLinks" => true,
                "maxAgeMs" => 86400000,
            ],
        ],
    ],
    tags: ["docs", "migration"],
    idempotencyKey: "docs-import-2026-07-31-v1",
);
var_dump($response);

The API reserves credits for accepted items at submission and returns 202. Invalid URLs and duplicate pairs of normalized URL and itemId are excluded. The same URL with two different item IDs counts as two items.

sample response

{
  "id": "batch_9f2c8a",
  "status": "queued",
  "mode": "scrape",
  "format": "markdown",
  "tags": ["docs", "migration"],
  "crawl": null,
  "input": {
    "reserved": 2,
    "reserved_is_ceiling": false,
    "submitted": 2,
    "duplicates": 0,
    "invalid": 0
  },
  "credits": {
    "reserved": 2
  },
  "created_at": "2026-07-31T12:00:00.000Z",
  "invalid_urls": [],
  "key_metadata": {
    "credits_consumed": 0,
    "credits_remaining": 500
  }
}

Retries with the same Idempotency-Key and request body return the original batch. Reusing the key with a different body returns 409 IDEMPOTENCY_KEY_CONFLICT. Keys can contain up to 200 characters.

Scraping options

Common options under input.data.options include:

Option	Description
`useMainContentOnly`	Remove navigation, headers, footers, and sidebars.
`includeHTML`	For Markdown batches, also include each page’s HTML beside its Markdown.
`includeSelectors` / `excludeSelectors`	Keep or remove matching CSS subtrees.
`maxAgeMs`	Reuse a cached scrape younger than this many milliseconds. Set `0` to scrape fresh.
`waitForMs`	Wait up to 15 seconds after browser load when rendering is needed.
`settleAnimations`	Wait briefly for CSS transitions and animations to settle.
`country`	Select the browser/proxy country.
`pdf`	Control PDF parsing, page range, and OCR. OCR is opt-in and billed at 1 credit per recovered page; skipped PDFs and image-only scans surface `PDF_SKIPPED` / `PDF_IMAGES_ONLY` in the page’s error record.

Markdown batches also support includeLinks, includeImages, and shortenBase64Images. See Submit a batch for the complete schema.

Crawl from a starting URL

Use a start_url source to follow links from one page. The starting URL is always included, even when regex does not match it. If the URL has no scheme, Context.dev reads it as HTTPS.

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-docs-crawl-v1" \
  -d '{
    "input": {
      "mode": "crawl",
      "data": {
        "format": "markdown",
        "source": {
          "type": "start_url",
          "url": "https://example.com/docs",
          "controls": {
            "maxUrls": 500,
            "maxDepth": 3,
            "followSubdomains": false,
            "regex": "^https://example\\.com/docs/"
          }
        },
        "options": {
          "useMainContentOnly": true
        }
      }
    },
    "tags": ["docs-crawl"]
  }'

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.submit({
  input: {
    mode: "crawl",
    data: {
      format: "markdown",
      source: {
        type: "start_url",
        url: "https://example.com/docs",
        controls: {
          maxUrls: 500,
          maxDepth: 3,
          followSubdomains: false,
          regex: "^https://example\\.com/docs/",
        },
      },
      options: {
        useMainContentOnly: true,
      },
    },
  },
  tags: ["docs-crawl"],
  "Idempotency-Key": "example-docs-crawl-v1",
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.submit(
    input={
        "mode": "crawl",
        "data": {
            "format": "markdown",
            "source": {
                "type": "start_url",
                "url": "https://example.com/docs",
                "controls": {
                    "max_urls": 500,
                    "max_depth": 3,
                    "follow_subdomains": False,
                    "regex": "^https://example\\.com/docs/",
                },
            },
            "options": {
                "use_main_content_only": True,
            },
        },
    },
    tags=["docs-crawl"],
    idempotency_key="example-docs-crawl-v1",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.submit(
  input: {
    mode: "crawl",
    data: {
      format: "markdown",
      source: {
        type: "start_url",
        url: "https://example.com/docs",
        controls: {
          max_urls: 500,
          max_depth: 3,
          follow_subdomains: false,
          regex: "^https://example\\.com/docs/",
        },
      },
      options: {
        use_main_content_only: true,
      },
    },
  },
  tags: ["docs-crawl"],
  idempotency_key: "example-docs-crawl-v1",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
		Input: contextdev.BatchSubmitParamsInputUnion{
			OfCrawl: &contextdev.BatchSubmitParamsInputCrawl{
				Mode: "crawl",
				Data: contextdev.BatchSubmitParamsInputCrawlDataUnion{
					OfMarkdown: &contextdev.BatchSubmitParamsInputCrawlDataMarkdown{
						Format: "markdown",
						Source: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceUnion{
							OfStartURL: &contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceStartURL{
								Type: "start_url",
								URL:  "https://example.com/docs",
								Controls: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceStartURLControls{
									MaxURLs:          contextdev.Int(500),
									MaxDepth:         contextdev.Int(3),
									FollowSubdomains: contextdev.Bool(false),
									Regex:            contextdev.String("^https://example\\.com/docs/"),
								},
							},
						},
						Options: contextdev.BatchSubmitParamsInputCrawlDataMarkdownOptions{
							UseMainContentOnly: contextdev.Bool(true),
						},
					},
				},
			},
		},
		Tags:           []string{"docs-crawl"},
		IdempotencyKey: contextdev.String("example-docs-crawl-v1"),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->submit(
    input: [
        "mode" => "crawl",
        "data" => [
            "format" => "markdown",
            "source" => [
                "type" => "start_url",
                "url" => "https://example.com/docs",
                "controls" => [
                    "maxURLs" => 500,
                    "maxDepth" => 3,
                    "followSubdomains" => false,
                    "regex" => "^https://example\\.com/docs/",
                ],
            ],
            "options" => [
                "useMainContentOnly" => true,
            ],
        ],
    ],
    tags: ["docs-crawl"],
    idempotencyKey: "example-docs-crawl-v1",
);
var_dump($response);

controls is optional. maxUrls defaults to 100, followSubdomains defaults to false, and an omitted maxDepth allows any depth until another control stops discovery.

Scrape URLs from a sitemap

Use a sitemap source to scrape URLs listed by a domain. The API accepts a bare domain or a full URL and reduces it to the domain.

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-sitemap-v1" \
  -d '{
    "input": {
      "mode": "crawl",
      "data": {
        "format": "html",
        "source": {
          "type": "sitemap",
          "domain": "example.com",
          "controls": {
            "maxUrls": 500,
            "regex": "^https://example\\.com/docs/"
          }
        }
      }
    }
  }'

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.submit({
  input: {
    mode: "crawl",
    data: {
      format: "html",
      source: {
        type: "sitemap",
        domain: "example.com",
        controls: {
          maxUrls: 500,
          regex: "^https://example\\.com/docs/",
        },
      },
    },
  },
  "Idempotency-Key": "example-sitemap-v1",
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.submit(
    input={
        "mode": "crawl",
        "data": {
            "format": "html",
            "source": {
                "type": "sitemap",
                "domain": "example.com",
                "controls": {
                    "max_urls": 500,
                    "regex": "^https://example\\.com/docs/",
                },
            },
        },
    },
    idempotency_key="example-sitemap-v1",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.submit(
  input: {
    mode: "crawl",
    data: {
      format: "html",
      source: {
        type: "sitemap",
        domain: "example.com",
        controls: {
          max_urls: 500,
          regex: "^https://example\\.com/docs/",
        },
      },
    },
  },
  idempotency_key: "example-sitemap-v1",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
		Input: contextdev.BatchSubmitParamsInputUnion{
			OfCrawl: &contextdev.BatchSubmitParamsInputCrawl{
				Mode: "crawl",
				Data: contextdev.BatchSubmitParamsInputCrawlDataUnion{
					OfHTML: &contextdev.BatchSubmitParamsInputCrawlDataHTML{
						Format: "html",
						Source: contextdev.BatchSubmitParamsInputCrawlDataHTMLSourceUnion{
							OfSitemap: &contextdev.BatchSubmitParamsInputCrawlDataHTMLSourceSitemap{
								Type:   "sitemap",
								Domain: "example.com",
								Controls: contextdev.BatchSubmitParamsInputCrawlDataHTMLSourceSitemapControls{
									MaxURLs: contextdev.Int(500),
									Regex:   contextdev.String("^https://example\\.com/docs/"),
								},
							},
						},
					},
				},
			},
		},
		IdempotencyKey: contextdev.String("example-sitemap-v1"),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->submit(
    input: [
        "mode" => "crawl",
        "data" => [
            "format" => "html",
            "source" => [
                "type" => "sitemap",
                "domain" => "example.com",
                "controls" => [
                    "maxURLs" => 500,
                    "regex" => "^https://example\\.com/docs/",
                ],
            ],
        ],
    ],
    idempotencyKey: "example-sitemap-v1",
);
var_dump($response);

A sitemap batch scrapes only URLs from the sitemap. It does not follow links from those pages, so sitemap controls support maxUrls and regex, but not maxDepth or followSubdomains.

Poll for completion

Save the id returned by submission, then replace batch_9f2c8a with that ID in the examples:

export BATCH_ID="batch_9f2c8a"

curl https://api.context.dev/v1/batch/$BATCH_ID \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.retrieve("batch_9f2c8a");
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.retrieve(
    "batch_9f2c8a",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.retrieve(
  "batch_9f2c8a",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Get(context.Background(), "batch_9f2c8a")
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->retrieve(
    "batch_9f2c8a",
);
var_dump($response);

Poll every 10 to 30 seconds while the status is queued, running, or cancelling. Stop when it becomes completed, cancelled, or failed. A webhook can reduce routine polling, but polling remains the fallback when delivery fails.

Field	Meaning
`progress.succeeded`	Pages scraped successfully.
`progress.failed`	Pages that could not be scraped.
`progress.pending`	Accepted pages not yet attempted. It is `0` when a batch completes.
`credits.reserved`	Credits debited when the batch was accepted.
`credits.refunded`	Credits returned for pages that did not succeed after the batch settles.
`credits.ocr_charged`	Additional credits charged for PDF pages recovered by OCR.
`credits.net`	Net cost: `reserved - refunded + ocr_charged`.
`page_errors`	Page failures grouped by error code.
`failure`	Batch-level failure detail when the batch itself fails.

A crawl can complete below maxUrls when no more eligible pages are reachable. The retrieve response repeats invalid_urls, but not webhook_secret. Store webhook_secret from the submit response immediately; it is returned only once.

Receive a webhook

Use the webhook object in a batch submission to choose a destination and retry policy:

{
  "webhook": {
    "url": "https://app.example.com/webhooks/context",
    "retry": { "delays_seconds": [30, 300, 3600] }
  }
}

Use retry: {} for the default schedule or retry: { "delays_seconds": [] } to disable automatic retries. Custom schedules accept up to 10 delays, each 1–86,400 seconds, totaling at most 72 hours. Omitting retry preserves one legacy best-effort attempt. The older webhookUrl field also remains a single-attempt option; do not combine it with webhook. Save webhook_secret from the submission response for a custom receiver; it is not included in later retrieve responses. Verify X-Context-Signature before using the payload. It has the form t=<unix>,v1=<hex hmac> and signs {timestamp}.{rawRequestBody} with HMAC-SHA256. Use a constant-time comparison and reject stale timestamps. The monitor verification example uses the same algorithm. After verification, deduplicate the event ID, enqueue work, and return a 2xx promptly. Inspect delivery history and attempts, then replay a retained event after repairing your receiver. Poll batches that remain unresolved in your system; Retrieve Batch is the authoritative status and provides fresh signed result URLs.

Send batch notifications to Slack

Create a Slack or GovSlack incoming webhook for the destination channel and use its URL as webhook.url. Terminal batch events arrive as formatted messages with a source link. Automatic retries and manual replay use the same Slack formatting. Keep the incoming-webhook URL in your secret store. Custom webhook destinations continue receiving the standard batch.completed, batch.failed, or batch.cancelled payloads.

Read the results

Once the batch is terminal, request a page of results:

curl -G https://api.context.dev/v1/batch/$BATCH_ID/results \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.getResults("batch_9f2c8a", {
  limit: 100,
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.get_results(
    "batch_9f2c8a",
    limit=100,
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.get_results(
  "batch_9f2c8a",
  limit: 100,
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.GetResults(context.Background(), "batch_9f2c8a", contextdev.BatchGetResultsParams{
		Limit: contextdev.Int(100),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->getResults(
    "batch_9f2c8a",
    limit: 100,
);
var_dump($response);

Successful records have status: "ok" plus http_status, final_url, metadata, and either markdown or html. Failed records have status: "error", error_code, and message. Scrape results echo any submitted itemId and meta.

sample response

{
  "data": [
    {
      "url": "https://docs.context.dev/introduction",
      "itemId": "docs-001",
      "meta": { "section": "getting-started" },
      "status": "ok",
      "http_status": 200,
      "final_url": "https://docs.context.dev/introduction",
      "markdown": "# Introduction\n\n...",
      "metadata": { "title": "Introduction" }
    },
    {
      "url": "https://example.com/missing",
      "status": "error",
      "error_code": "NOT_FOUND",
      "message": "Page returned 404"
    }
  ],
  "has_more": true,
  "next_cursor": "...",
  "key_metadata": {
    "credits_consumed": 0,
    "credits_remaining": 498
  }
}

A page can close before limit to keep its payload under approximately 8 MB. Continue based on has_more and next_cursor, not the number of records. Replace CURSOR_FROM_PREVIOUS_RESPONSE with the returned cursor, or set NEXT_CURSOR for cURL:

curl -G https://api.context.dev/v1/batch/$BATCH_ID/results \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100" \
  --data-urlencode "cursor=$NEXT_CURSOR"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.getResults("batch_9f2c8a", {
  limit: 100,
  cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.get_results(
    "batch_9f2c8a",
    limit=100,
    cursor="CURSOR_FROM_PREVIOUS_RESPONSE",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.get_results(
  "batch_9f2c8a",
  limit: 100,
  cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.GetResults(context.Background(), "batch_9f2c8a", contextdev.BatchGetResultsParams{
		Limit:  contextdev.Int(100),
		Cursor: contextdev.String("CURSOR_FROM_PREVIOUS_RESPONSE"),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->getResults(
    "batch_9f2c8a",
    limit: 100,
    cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
);
var_dump($response);

For large result sets, retrieve the batch and download the signed URLs in results.files instead. Each file is gzipped NDJSON. The response includes results.expires_at; retrieve the batch again to obtain fresh links after they expire.

Find previous batches

List batches newest first and combine status, search, and tag filters:

curl -G https://api.context.dev/v1/batch/list \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "status=completed" \
  --data-urlencode "q=docs" \
  --data-urlencode "search_type=prefix" \
  --data-urlencode "tags=docs,migration" \
  --data-urlencode "limit=50"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.list({
  status: "completed",
  q: "docs",
  search_type: "prefix",
  tags: "docs,migration",
  limit: 50,
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.list(
    status="completed",
    q="docs",
    search_type="prefix",
    tags="docs,migration",
    limit=50,
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.list(
  status: "completed",
  q: "docs",
  search_type: "prefix",
  tags: "docs,migration",
  limit: 50,
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.List(context.Background(), contextdev.BatchListParams{
		Status:     "completed",
		Q:          contextdev.String("docs"),
		SearchType: "prefix",
		Tags:       contextdev.String("docs,migration"),
		Limit:      contextdev.Int(50),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->list(
    status: "completed",
    q: "docs",
    searchType: "prefix",
    tags: "docs,migration",
    limit: 50,
);
var_dump($response);

q matches batch IDs, crawl sources, and tags. The tags filter matches batches containing any supplied tag. Continue with next_cursor when has_more is true.

Cancel or delete a batch

Cancellation stops new pages from starting. Pages already in progress finish before the batch reaches cancelled, and unused reserved credits are released.

curl -X POST https://api.context.dev/v1/batch/$BATCH_ID/cancel \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.cancel("batch_9f2c8a");
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.cancel(
    "batch_9f2c8a",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.cancel(
  "batch_9f2c8a",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Cancel(context.Background(), "batch_9f2c8a")
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->cancel(
    "batch_9f2c8a",
);
var_dump($response);

After a batch reaches completed, cancelled, or failed, you can permanently delete its metadata and stored results:

curl -X DELETE https://api.context.dev/v1/batch/$BATCH_ID \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.delete("batch_9f2c8a");
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.delete(
    "batch_9f2c8a",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.delete(
  "batch_9f2c8a",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Delete(context.Background(), "batch_9f2c8a")
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->delete(
    "batch_9f2c8a",
);
var_dump($response);

Handle errors

Batch management has a separate 1,000-unit-per-minute organization limit. A submission uses 50 units; each read, poll, cancellation, or deletion uses 1. See Batch rate limits.

Status	Error	What to do
400	Invalid request or unusable input	Check the discriminated input shape, URL count, and strict field names.
401	Authentication or balance error	Check the API key and available credits.
403	`BATCH_LIMIT_EXCEEDED`	Wait for an active batch to finish before submitting another.
409	`IDEMPOTENCY_KEY_CONFLICT`	Use the original body or choose a new idempotency key.
409	`BATCH_NOT_COMPLETED`	Wait for a terminal status before reading results.
409	Batch cannot be cancelled or deleted	Retrieve it and act based on its current status.
429	`RATE_LIMITED`	Wait for the rate-limit window and retry with backoff.
500	Input could not be staged or queued	Retry with the same idempotency key so a duplicate is not created.

Next steps

Build with an agent

Add batch scraping to your app

Open in Cursor

Submit a batch

Full request schema, examples, and responses.

Batch results

Paginated success and error records.

Scrape a webpage

Use synchronous scraping for one page at a time.

Best practices

Authentication, retries, caching, and error handling.

Get started

Web data

Brand data

Automation

Account and billing

Production

Prerequisites

Choose an input

Submit a fixed URL list

Poll for completion

Receive a webhook

Send batch notifications to Slack

Read the results

Find previous batches

Cancel or delete a batch

Handle errors

Next steps

Submit a batch

Batch results

Scrape a webpage

Best practices

​Prerequisites

​Choose an input

​Submit a fixed URL list

​Poll for completion

​Receive a webhook

​Send batch notifications to Slack

​Read the results

​Find previous batches

​Cancel or delete a batch

​Handle errors

​Next steps

Submit a batch

Batch results

Scrape a webpage

Best practices

Prerequisites

Choose an input

Submit a fixed URL list

Poll for completion

Receive a webhook

Send batch notifications to Slack

Read the results

Find previous batches

Cancel or delete a batch

Handle errors

Next steps