> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Sync

> Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.

`POST /web/crawl` follows links from a starting URL and returns page Markdown in the same HTTP response. There is no background job to poll.

<a id="choose-crawl-or-batch" />

<Info>
  Use **Crawl Sync** for small crawls under 500 pages when you need results in one response. Choose [Crawl Async (25K pages)](/guides/crawl-website-async) for larger sites, background processing, or crawls that outlast the sync time budget.
</Info>

Crawls cost 1 credit per successfully scraped page, plus 1 credit per PDF page recovered by OCR when enabled. Set `maxPages` to bound the crawl; PDF OCR can add to that budget.

## Prerequisites

Export an API key from the [dashboard](https://context.dev/dashboard):

```bash theme={null}
export CONTEXT_DEV_API_KEY="ctxt_secret_..."
```

## Crawl a bounded section

This request starts at the Context.dev documentation, follows links one hop away, and collects up to three pages.

<CodeGroup>
  ```bash cURL theme={null}
  curl https://api.context.dev/v1/web/crawl \
    --request POST \
    --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "url": "https://docs.context.dev/introduction",
      "maxPages": 3,
      "maxDepth": 1,
      "useMainContentOnly": true
    }'
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({
    apiKey: process.env.CONTEXT_DEV_API_KEY,
  });

  const crawl = await client.web.webCrawlMd({
    url: "https://docs.context.dev/introduction",
    maxPages: 3,
    maxDepth: 1,
    useMainContentOnly: true,
  });

  for (const page of crawl.results) {
    console.log(page.metadata.url, page.metadata.success);
  }
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  crawl = client.web.web_crawl_md(
      url="https://docs.context.dev/introduction",
      max_pages=3,
      max_depth=1,
      use_main_content_only=True,
  )

  for page in crawl.results:
      print(page.metadata.url, page.metadata.success)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  crawl = client.web.web_crawl_md(
    url: "https://docs.context.dev/introduction",
    max_pages: 3,
    max_depth: 1,
    use_main_content_only: true,
  )

  crawl.results.each do |page|
    puts "#{page.metadata.url}: #{page.metadata.success}"
  end
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      contextdev "github.com/context-dot-dev/context-go-sdk/v2"
      "github.com/context-dot-dev/context-go-sdk/v2/option"
      "github.com/context-dot-dev/context-go-sdk/v2/packages/param"
  )

  func main() {
      client := contextdev.NewClient(
          option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
      )

      crawl, err := client.Web.WebCrawlMd(context.Background(), contextdev.WebWebCrawlMdParams{
          URL: "https://docs.context.dev/introduction",
          MaxPages: param.NewOpt[int64](3),
          MaxDepth: param.NewOpt[int64](1),
          UseMainContentOnly: param.NewOpt(true),
      })
      if err != nil {
          panic(err)
      }

      for _, page := range crawl.Results {
          fmt.Println(page.Metadata.URL, page.Metadata.Success)
      }
  }
  ```

  ```php PHP theme={null}
  <?php

  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $crawl = $client->web->webCrawlMd(
      url: 'https://docs.context.dev/introduction',
      maxPages: 3,
      maxDepth: 1,
      useMainContentOnly: true,
  );

  foreach ($crawl->results as $page) {
      echo $page->metadata->url, ': ',
          $page->metadata->success ? 'success' : 'failed', PHP_EOL;
  }
  ```
</CodeGroup>

## Read the result

The response separates page results from crawl-level totals:

```json theme={null}
{
  "results": [
    {
      "markdown": "# Introduction\n\nContext.dev gives software...",
      "metadata": {
        "url": "https://docs.context.dev/introduction",
        "title": "Introduction",
        "crawlDepth": 0,
        "statusCode": 200,
        "success": true
      }
    }
  ],
  "metadata": {
    "numUrls": 1,
    "maxCrawlDepth": 0,
    "numSucceeded": 1,
    "numFailed": 0,
    "numSkipped": 0
  },
  "cache_metadata": {
    "status": "miss",
    "age_ms": 0
  }
}
```

Check `results[].metadata.success` before using a page's Markdown. A successful HTTP response for the crawl can contain page-level failures or skipped pages.

## Control crawl coverage

| Goal                          | Setting                 | Notes                                                                                                              |
| ----------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Cap spend                     | `maxPages`              | Defaults to 100; accepted range is 1 to 500.                                                                       |
| Limit link distance           | `maxDepth`              | `0` keeps only the starting page; `1` follows one hop.                                                             |
| Stay inside one path          | `urlRegex`              | Only matching URLs are followed and scraped. Test the expression against representative URLs before a large crawl. |
| Include child hosts           | `followSubdomains=true` | Allows links such as `docs.example.com` from `example.com`.                                                        |
| Stop after a soft time budget | `stopAfterMs`           | Returns collected pages after 10 to 110 seconds; defaults to 80 seconds.                                           |
| Abort the entire request      | `timeoutMS`             | Returns `408` when the hard timeout is exceeded.                                                                   |

Content controls apply to each crawled page, including `includeSelectors`, `excludeSelectors`, `includeFrames`, `country`, and PDF parsing options.

## Production behavior

* The API accepts up to 500 pages, but `maxPages` is a ceiling, not a guarantee that the crawl finishes within its time budget. Use [Crawl Async](/guides/crawl-website-async) when the job needs to continue in the background.
* `maxAgeMs` applies to each page and defaults to one day. Set it to `0` when every page must be fetched again.
* A soft `stopAfterMs` can produce a useful partial result. A hard `timeoutMS` produces a `408` instead.
* `urlRegex` is both a relevance control and a cost control. Test it against expected URLs before raising `maxPages`.
* `zdr=enabled` bypasses shared caches and retained content logs when [Zero Data Retention](/optimization/zero-data-retention) is enabled for your organization.

## Next steps

<CardGroup cols={2}>
  <Card title="Crawl Async" icon="layer-group" href="/guides/crawl-website-async">
    Crawl up to 25,000 pages in the background and retrieve results later.
  </Card>

  <Card title="Crawl API reference" icon="code" href="/api-reference/web-scraping/crawl">
    Review the full crawl request and response contract.
  </Card>

  <Card title="Discover website URLs" icon="sitemap" href="/guides/discover-website-urls">
    Inspect sitemap URLs before estimating a large crawl.
  </Card>

  <Card title="Build a RAG pipeline" icon="database" href="/use-cases/build-rag-from-websites">
    Turn the returned Markdown into a searchable knowledge base.
  </Card>
</CardGroup>