Crawl Async

Use async crawling for large sites or jobs that should run in the background, up to 25,000 pages. For fewer than 500 pages when you need results in one response, use Crawl Sync.

Submit a starting URL to POST /batch/submit. The API discovers linked pages and processes them as a background job. Save the batch ID, poll for completion, then read or download the results.

Submit a crawl

Choose cURL or install an SDK for your language. Each example creates its own client. Export an API key from the dashboard:

export CONTEXT_DEV_API_KEY="ctxt_secret_..."

This request follows links within the documentation site, up to three hops from the starting page. It caps the job at 100 pages so you can check coverage before increasing the limit.

curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-crawl-v1" \
  -d '{
    "input": {
      "mode": "crawl",
      "data": {
        "format": "markdown",
        "source": {
          "type": "start_url",
          "url": "https://docs.context.dev/introduction",
          "controls": {
            "maxUrls": 100,
            "maxDepth": 3,
            "followSubdomains": false,
            "regex": "^https://docs\\.context\\.dev/"
          }
        },
        "options": {
          "useMainContentOnly": true
        }
      }
    },
    "tags": ["docs-crawl"]
  }'

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.submit({
  input: {
    mode: "crawl",
    data: {
      format: "markdown",
      source: {
        type: "start_url",
        url: "https://docs.context.dev/introduction",
        controls: {
          maxUrls: 100,
          maxDepth: 3,
          followSubdomains: false,
          regex: "^https://docs\\.context\\.dev/",
        },
      },
      options: {
        useMainContentOnly: true,
      },
    },
  },
  tags: ["docs-crawl"],
  "Idempotency-Key": "docs-crawl-v1",
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.submit(
    input={
        "mode": "crawl",
        "data": {
            "format": "markdown",
            "source": {
                "type": "start_url",
                "url": "https://docs.context.dev/introduction",
                "controls": {
                    "max_urls": 100,
                    "max_depth": 3,
                    "follow_subdomains": False,
                    "regex": "^https://docs\\.context\\.dev/",
                },
            },
            "options": {
                "use_main_content_only": True,
            },
        },
    },
    tags=["docs-crawl"],
    idempotency_key="docs-crawl-v1",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.submit(
  input: {
    mode: "crawl",
    data: {
      format: "markdown",
      source: {
        type: "start_url",
        url: "https://docs.context.dev/introduction",
        controls: {
          max_urls: 100,
          max_depth: 3,
          follow_subdomains: false,
          regex: "^https://docs\\.context\\.dev/",
        },
      },
      options: {
        use_main_content_only: true,
      },
    },
  },
  tags: ["docs-crawl"],
  idempotency_key: "docs-crawl-v1",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
		Input: contextdev.BatchSubmitParamsInputUnion{
			OfCrawl: &contextdev.BatchSubmitParamsInputCrawl{
				Mode: "crawl",
				Data: contextdev.BatchSubmitParamsInputCrawlDataUnion{
					OfMarkdown: &contextdev.BatchSubmitParamsInputCrawlDataMarkdown{
						Format: "markdown",
						Source: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceUnion{
							OfStartURL: &contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceStartURL{
								Type: "start_url",
								URL:  "https://docs.context.dev/introduction",
								Controls: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceStartURLControls{
									MaxURLs:          contextdev.Int(100),
									MaxDepth:         contextdev.Int(3),
									FollowSubdomains: contextdev.Bool(false),
									Regex:            contextdev.String("^https://docs\\.context\\.dev/"),
								},
							},
						},
						Options: contextdev.BatchSubmitParamsInputCrawlDataMarkdownOptions{
							UseMainContentOnly: contextdev.Bool(true),
						},
					},
				},
			},
		},
		Tags:           []string{"docs-crawl"},
		IdempotencyKey: contextdev.String("docs-crawl-v1"),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->submit(
    input: [
        "mode" => "crawl",
        "data" => [
            "format" => "markdown",
            "source" => [
                "type" => "start_url",
                "url" => "https://docs.context.dev/introduction",
                "controls" => [
                    "maxURLs" => 100,
                    "maxDepth" => 3,
                    "followSubdomains" => false,
                    "regex" => "^https://docs\\.context\\.dev/",
                ],
            ],
            "options" => [
                "useMainContentOnly" => true,
            ],
        ],
    ],
    tags: ["docs-crawl"],
    idempotencyKey: "docs-crawl-v1",
);
var_dump($response);

An accepted request returns 202 with status: "queued" and an id. Save that ID before starting another job. Submission reserves credits immediately; see costs and limits before raising maxUrls. Choose a new Idempotency-Key for each new crawl. Reuse the same key and body when retrying a submission to recover the original batch instead of creating a duplicate.

Control page discovery

Set these fields under input.data.source.controls:

Setting	Behavior
`maxUrls`	Maximum pages to fetch, from 1 to 25,000. Defaults to 100; the crawl can finish below this cap.
`maxDepth`	Maximum link depth, from 0 to 50. The starting page is depth 0. Omit it for no depth limit.
`regex`	RE2 pattern for eligible URLs, up to 256 characters. The starting URL is always included, even if it does not match.
`followSubdomains`	Follow links to subdomains. Defaults to `false`.

Use input.data.format: "html" for HTML instead of Markdown. Per-page controls belong in input.data.options, including content selectors, rendering waits, and PDF parsing. maxAgeMs defaults to one day; set it to 0 to fetch pages fresh.

Start from a sitemap

To scrape the URLs listed in a sitemap, submit a sitemap source:

curl https://api.context.dev/v1/batch/submit \
  --request POST \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
  "input": {
    "mode": "crawl",
    "data": {
      "format": "markdown",
      "source": {
        "type": "sitemap",
        "domain": "example.com",
        "controls": {
          "maxUrls": 100,
          "regex": "^https://example\\.com/docs/"
        }
      }
    }
  }
}'

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.submit({
  input: {
    mode: "crawl",
    data: {
      format: "markdown",
      source: {
        type: "sitemap",
        domain: "example.com",
        controls: {
          maxUrls: 100,
          regex: "^https://example\\.com/docs/",
        },
      },
    },
  },
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.submit(
    input={
        "mode": "crawl",
        "data": {
            "format": "markdown",
            "source": {
                "type": "sitemap",
                "domain": "example.com",
                "controls": {
                    "max_urls": 100,
                    "regex": "^https://example\\.com/docs/",
                },
            },
        },
    },
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.submit(
  input: {
    mode: "crawl",
    data: {
      format: "markdown",
      source: {
        type: "sitemap",
        domain: "example.com",
        controls: {
          max_urls: 100,
          regex: "^https://example\\.com/docs/",
        },
      },
    },
  },
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
		Input: contextdev.BatchSubmitParamsInputUnion{
			OfCrawl: &contextdev.BatchSubmitParamsInputCrawl{
				Mode: "crawl",
				Data: contextdev.BatchSubmitParamsInputCrawlDataUnion{
					OfMarkdown: &contextdev.BatchSubmitParamsInputCrawlDataMarkdown{
						Format: "markdown",
						Source: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceUnion{
							OfSitemap: &contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceSitemap{
								Type:   "sitemap",
								Domain: "example.com",
								Controls: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceSitemapControls{
									MaxURLs: contextdev.Int(100),
									Regex:   contextdev.String("^https://example\\.com/docs/"),
								},
							},
						},
					},
				},
			},
		},
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->submit(
    input: [
        "mode" => "crawl",
        "data" => [
            "format" => "markdown",
            "source" => [
                "type" => "sitemap",
                "domain" => "example.com",
                "controls" => [
                    "maxURLs" => 100,
                    "regex" => "^https://example\\.com/docs/",
                ],
            ],
        ],
    ],
);
var_dump($response);

Sitemap batches scrape matching listed URLs without following links from those pages. Only maxUrls and regex are supported in their controls; omit maxDepth and followSubdomains. A full URL in domain is reduced to its domain.

Track progress

Replace batch_9f2c8a with the ID returned by submission. For cURL, save it in BATCH_ID:

export BATCH_ID="batch_9f2c8a"

curl "https://api.context.dev/v1/batch/$BATCH_ID" \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.retrieve("batch_9f2c8a");
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.retrieve(
    "batch_9f2c8a",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.retrieve(
  "batch_9f2c8a",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.Get(context.Background(), "batch_9f2c8a")
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->retrieve(
    "batch_9f2c8a",
);
var_dump($response);

Poll every 10 to 30 seconds while status is queued, running, or cancelling. Stop when it reaches completed, cancelled, or failed. Use the status, not a percentage of maxUrls, to decide whether the job has finished. progress.succeeded and progress.failed count page outcomes. page_errors groups page failures by code; failure describes a batch-level failure. A completed batch can still contain failed pages. You can also add webhookUrl to the submit body. Completion webhooks are attempted once, so keep polling as a fallback and save the signing secret returned at submission.

Read the results

After the batch reaches a final status, request its results as JSON:

curl -G "https://api.context.dev/v1/batch/$BATCH_ID/results" \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.getResults("batch_9f2c8a", {
  limit: 100,
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.get_results(
    "batch_9f2c8a",
    limit=100,
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.get_results(
  "batch_9f2c8a",
  limit: 100,
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.GetResults(context.Background(), "batch_9f2c8a", contextdev.BatchGetResultsParams{
		Limit: contextdev.Int(100),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->getResults(
    "batch_9f2c8a",
    limit: 100,
);
var_dump($response);

Each record describes one page. Check status before reading its content:

sample response

{
  "data": [
    {
      "url": "https://docs.context.dev/introduction",
      "status": "ok",
      "http_status": 200,
      "final_url": "https://docs.context.dev/introduction",
      "markdown": "# Introduction\n\n...",
      "metadata": {
        "sourceUrl": "https://docs.context.dev/introduction",
        "finalUrl": "https://docs.context.dev/introduction",
        "title": "Introduction"
      },
      "cache_metadata": { "status": "miss", "age_ms": 0 }
    },
    {
      "url": "https://docs.context.dev/missing-page",
      "status": "error",
      "error_code": "NOT_FOUND",
      "message": "Page returned 404"
    }
  ],
  "has_more": true,
  "next_cursor": "..."
}

While has_more is true, pass the returned next_cursor as cursor. Replace CURSOR_FROM_PREVIOUS_RESPONSE in the SDK examples, or set NEXT_CURSOR for cURL:

curl -G "https://api.context.dev/v1/batch/$BATCH_ID/results" \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "limit=100" \
  --data-urlencode "cursor=$NEXT_CURSOR"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.batch.getResults("batch_9f2c8a", {
  limit: 100,
  cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
});
console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.batch.get_results(
    "batch_9f2c8a",
    limit=100,
    cursor="CURSOR_FROM_PREVIOUS_RESPONSE",
)
print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.batch.get_results(
  "batch_9f2c8a",
  limit: 100,
  cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
)
puts response.inspect

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

	response, err := client.Batch.GetResults(context.Background(), "batch_9f2c8a", contextdev.BatchGetResultsParams{
		Limit:  contextdev.Int(100),
		Cursor: contextdev.String("CURSOR_FROM_PREVIOUS_RESPONSE"),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->batch->getResults(
    "batch_9f2c8a",
    limit: 100,
    cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
);
var_dump($response);

limit accepts 1 to 100 records and defaults to 25. A page can close early to stay under approximately 8 MB, so follow the cursor even when fewer than 100 records arrive. For large imports, use the signed URLs in the retrieved batch’s results.files to download gzipped NDJSON. Set RESULT_URL to a file’s url, then stream its records:

curl --fail --location "$RESULT_URL" | gzip --decompress

Download each file. Links expire at results.expires_at; retrieve the batch again for fresh links. File order is not guaranteed.

Costs and limits

Each successfully scraped page costs 1 credit. A start-URL crawl reserves its page budget upfront: input.reserved_is_ceiling is true because the reachable page count is not yet known. A sitemap batch reserves for the exact accepted URL count instead. Credits for pages that do not succeed, including unused crawl capacity, are refunded when the batch settles. If PDF OCR is enabled, each recovered PDF page costs 1 additional credit. OCR is off by default. Read credits.net after settlement for the final cost: reserved - refunded + ocr_charged. Batch management uses a separate rate-limit bucket. Submission consumes 50 of its 1,000 units per minute; each poll, results request, or cancellation consumes 1. These management calls cost zero API credits, separate from the page credits reserved at submission.

Handle failures

Retry only the failed pages when practical, using a fixed URL batch. Keep successful content and inspect each failed record’s error_code and message before retrying.

Response	What to do
`400`	Check the request fields, page limits, and source URL.
`403 BATCH_LIMIT_EXCEEDED`	Wait for an active batch to finish before submitting another.
`409 IDEMPOTENCY_KEY_CONFLICT`	Reuse the original request body or choose a new key for a new job.
`409 BATCH_NOT_COMPLETED`	Keep polling before requesting results.
`429 RATE_LIMITED`	Honor `Retry-After` and retry with backoff.
`500` at submission	Retry with the same idempotency key and body.

To stop a crawl, call POST /batch/{batch_id}/cancel. This prevents new pages from starting; pages already in progress finish. Continue polling until the job settles and unused credits are refunded.

Next steps

Crawl Sync

Collect a smaller site in one synchronous response.

Submit a batch

Review the full request schema and per-page options.

Build a RAG pipeline

Turn crawled Markdown into a searchable knowledge base.

Get started

Web data

Brand data

Automation

Production

Submit a crawl

Control page discovery

Track progress

Read the results

Costs and limits

Handle failures

Next steps

Crawl Sync

Submit a batch

Build a RAG pipeline

​Submit a crawl

​Control page discovery

​Track progress

​Read the results

​Costs and limits

​Handle failures

​Next steps

Crawl Sync

Submit a batch

Build a RAG pipeline

Submit a crawl

Control page discovery

Track progress

Read the results

Costs and limits

Handle failures

Next steps