Crawl Sync

POST /web/crawl follows links from a starting URL and returns page Markdown in the same HTTP response. There is no background job to poll.

Use Crawl Sync for small crawls under 500 pages when you need results in one response. Choose Crawl Async (25K pages) for larger sites, background processing, or crawls that outlast the sync time budget.

Crawls cost 1 credit per successfully scraped page, plus 1 credit per PDF page recovered by OCR when enabled. Set maxPages to bound the crawl; PDF OCR can add to that budget.

Prerequisites

Export an API key from the dashboard:

export CONTEXT_DEV_API_KEY="ctxt_secret_..."

Crawl a bounded section

This request starts at the Context.dev documentation, follows links one hop away, and collects up to three pages.

curl https://api.context.dev/v1/web/crawl \
  --request POST \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "url": "https://docs.context.dev/introduction",
    "maxPages": 3,
    "maxDepth": 1,
    "useMainContentOnly": true
  }'

import ContextDev from "context.dev";

const client = new ContextDev({
  apiKey: process.env.CONTEXT_DEV_API_KEY,
});

const crawl = await client.web.webCrawlMd({
  url: "https://docs.context.dev/introduction",
  maxPages: 3,
  maxDepth: 1,
  useMainContentOnly: true,
});

for (const page of crawl.results) {
  console.log(page.metadata.url, page.metadata.success);
}

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

crawl = client.web.web_crawl_md(
    url="https://docs.context.dev/introduction",
    max_pages=3,
    max_depth=1,
    use_main_content_only=True,
)

for page in crawl.results:
    print(page.metadata.url, page.metadata.success)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

crawl = client.web.web_crawl_md(
  url: "https://docs.context.dev/introduction",
  max_pages: 3,
  max_depth: 1,
  use_main_content_only: true,
)

crawl.results.each do |page|
  puts "#{page.metadata.url}: #{page.metadata.success}"
end

package main

import (
    "context"
    "fmt"
    "os"

    contextdev "github.com/context-dot-dev/context-go-sdk/v2"
    "github.com/context-dot-dev/context-go-sdk/v2/option"
    "github.com/context-dot-dev/context-go-sdk/v2/packages/param"
)

func main() {
    client := contextdev.NewClient(
        option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
    )

    crawl, err := client.Web.WebCrawlMd(context.Background(), contextdev.WebWebCrawlMdParams{
        URL: "https://docs.context.dev/introduction",
        MaxPages: param.NewOpt[int64](3),
        MaxDepth: param.NewOpt[int64](1),
        UseMainContentOnly: param.NewOpt(true),
    })
    if err != nil {
        panic(err)
    }

    for _, page := range crawl.Results {
        fmt.Println(page.Metadata.URL, page.Metadata.Success)
    }
}

<?php

require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$crawl = $client->web->webCrawlMd(
    url: 'https://docs.context.dev/introduction',
    maxPages: 3,
    maxDepth: 1,
    useMainContentOnly: true,
);

foreach ($crawl->results as $page) {
    echo $page->metadata->url, ': ',
        $page->metadata->success ? 'success' : 'failed', PHP_EOL;
}

Read the result

The response separates page results from crawl-level totals:

{
  "results": [
    {
      "markdown": "# Introduction\n\nContext.dev gives software...",
      "metadata": {
        "url": "https://docs.context.dev/introduction",
        "title": "Introduction",
        "crawlDepth": 0,
        "statusCode": 200,
        "success": true
      }
    }
  ],
  "metadata": {
    "numUrls": 1,
    "maxCrawlDepth": 0,
    "numSucceeded": 1,
    "numFailed": 0,
    "numSkipped": 0
  },
  "cache_metadata": {
    "status": "miss",
    "age_ms": 0
  }
}

Check results[].metadata.success before using a page’s Markdown. A successful HTTP response for the crawl can contain page-level failures or skipped pages.

Control crawl coverage

Goal	Setting	Notes
Cap spend	`maxPages`	Defaults to 100; accepted range is 1 to 500.
Limit link distance	`maxDepth`	`0` keeps only the starting page; `1` follows one hop.
Stay inside one path	`urlRegex`	Only matching URLs are followed and scraped. Test the expression against representative URLs before a large crawl.
Include child hosts	`followSubdomains=true`	Allows links such as `docs.example.com` from `example.com`.
Stop after a soft time budget	`stopAfterMs`	Returns collected pages after 10 to 110 seconds; defaults to 80 seconds.
Abort the entire request	`timeoutMS`	Returns `408` when the hard timeout is exceeded.

Content controls apply to each crawled page, including includeSelectors, excludeSelectors, includeFrames, country, and PDF parsing options.

Production behavior

The API accepts up to 500 pages, but maxPages is a ceiling, not a guarantee that the crawl finishes within its time budget. Use Crawl Async when the job needs to continue in the background.
maxAgeMs applies to each page and defaults to one day. Set it to 0 when every page must be fetched again.
A soft stopAfterMs can produce a useful partial result. A hard timeoutMS produces a 408 instead.
urlRegex is both a relevance control and a cost control. Test it against expected URLs before raising maxPages.
zdr=enabled bypasses shared caches and retained content logs when Zero Data Retention is enabled for your organization.

Next steps

Crawl Async

Crawl up to 25,000 pages in the background and retrieve results later.

Crawl API reference

Review the full crawl request and response contract.

Discover website URLs

Inspect sitemap URLs before estimating a large crawl.

Build a RAG pipeline

Turn the returned Markdown into a searchable knowledge base.

Get started

Web data

Brand data

Automation

Production

Prerequisites

Crawl a bounded section

Read the result

Control crawl coverage

Production behavior

Next steps

Crawl Async

Crawl API reference

Discover website URLs

Build a RAG pipeline

​Prerequisites

​Crawl a bounded section

​Read the result

​Control crawl coverage

​Production behavior

​Next steps

Crawl Async

Crawl API reference

Discover website URLs

Build a RAG pipeline

Prerequisites

Crawl a bounded section

Read the result

Control crawl coverage

Production behavior

Next steps