> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Websites

> Scrape any URL to clean Markdown or HTML, crawl a whole domain, list sitemap URLs, or extract images. Handles PDF, DOCX, XLSX, PPTX, and CSV natively.

Context.dev's Web APIs can:

* Turn a webpage into clean markdown/HTML
* Crawl an entire website and save every page as markdown
* Get all webpages under a domain
* Extract every image on a webpage

Context.dev's scrapers automatically switch to a different web-proxy when they get blocked by bot protection or geo-restrictions.

<Prompt description="Integrate Context.dev's scraping endpoints in your app" icon="sparkles" actions={["copy", "cursor"]}>
  I'm integrating Context.dev's web scraping endpoints into my app. Help me:

  1. Install the official SDK for my language: `context.dev` on npm (TypeScript), `context.dev` on PyPI (Python), `context.dev` on RubyGems (Ruby), `context-dev/context-dev-php` on Packagist (PHP), or `github.com/context-dot-dev/context-go-sdk` (Go). Or call the raw HTTP API with `curl` / `fetch`.
  2. Read the API key from the `CONTEXT_DEV_API_KEY` environment variable. Never hardcode it. Authentication is a bearer token on every request.
  3. Call `/web/scrape/markdown` to turn a URL into clean Markdown (pass `useMainContentOnly: true` to strip nav, footer, and chrome).
  4. Call `/web/scrape/html` for raw rendered HTML, `/web/scrape/images` for an image manifest, `/web/scrape/sitemap` to list a domain's URLs, and `/web/crawl` to walk a whole domain with `maxPages` and `maxDepth` caps.
  5. Wrap calls with exponential backoff for transient failures. Proxy escalation is automatic, so most retries succeed on a different IP pool.

  Docs: [https://docs.context.dev/guides/scrape-websites-to-markdown](https://docs.context.dev/guides/scrape-websites-to-markdown)
</Prompt>

## Prerequisites

* **A Context.dev API key.** Sign up at [context.dev/signup](https://context.dev/signup), copy the key from the [dashboard](https://context.dev/dashboard) (prefix `ctxt_secret_`), and export it:

  ```bash theme={null}
  export CONTEXT_DEV_API_KEY="ctxt_secret_..."
  ```

* **An SDK (optional).** Install for your language, or skip the install and call directly with `curl`:

  <CodeGroup>
    ```bash TypeScript theme={null}
    npm install context.dev
    ```

    ```bash Python theme={null}
    pip install context.dev
    ```

    ```bash Ruby theme={null}
    gem install context.dev
    ```

    ```bash Go theme={null}
    go get github.com/context-dot-dev/context-go-sdk
    ```

    ```bash PHP theme={null}
    composer require context-dev/context-dev-php
    ```
  </CodeGroup>

## Scrape a single page to Markdown

`GET /web/scrape/markdown` scrapes any URL into LLM-ready GitHub Flavored Markdown. Bot protection and geo-blocks are handled by automatic proxy escalation; pass `useMainContentOnly: true` to drop nav, footer, sidebars, and other chrome.

The endpoint transparently handles HTML, XML, JSON, text, Markdown, SVG, PDF, DOCX, DOC, XLSX, XLS, PPTX, PPT, and CSV. Excel workbooks come back as one GFM table per sheet (with each sheet name as an `##` heading); PowerPoint decks come back as slide-structured markdown (one `## Slide N: Title` per slide, followed by body text, tables, and speaker notes); CSV files come back as a single GFM table, with any comma-less leading line surfaced as text above it. Unsupported formats (images, media, archives) return a `415`.

<CodeGroup>
  ```bash cURL theme={null}
  curl -G https://api.context.dev/v1/web/scrape/markdown \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --data-urlencode "url=https://example.com" \
    --data-urlencode "useMainContentOnly=true"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.web.webScrapeMd({
    url: "https://example.com",
    useMainContentOnly: true,
  });

  console.log(response.markdown);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.web.web_scrape_md(
      url="https://example.com",
      use_main_content_only=True,
  )

  print(response.markdown)
  ```

  ```ruby Ruby theme={null}
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.web.web_scrape_md(
    url: "https://example.com",
    use_main_content_only: true,
  )

  puts response.markdown
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      contextdev "github.com/context-dot-dev/context-go-sdk"
      "github.com/context-dot-dev/context-go-sdk/option"
      "github.com/context-dot-dev/context-go-sdk/packages/param"
  )

  func main() {
      client := contextdev.NewClient(
          option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
      )

      response, err := client.Web.WebScrapeMd(context.TODO(), contextdev.WebWebScrapeMdParams{
          URL:                "https://example.com",
          UseMainContentOnly: param.NewOpt(true),
      })
      if err != nil {
          panic(err)
      }

      fmt.Println(response.Markdown)
  }
  ```

  ```php PHP theme={null}
  <?php

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->web->webScrapeMd(
      url: 'https://example.com',
      useMainContentOnly: true,
  );

  echo $response->markdown, PHP_EOL;
  ```
</CodeGroup>

<Badge color="blue" icon="coins">1 credit per call</Badge>

The connection stays open while the page is fetched and converted, so there's no need to poll. Repeated calls for the same URL within `maxAgeMs` return the cached scrape.

### Request Parameters

| Parameter             | Type         | Default                 | Description                                                                                                                                                                                                                          |
| --------------------- | ------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url`                 | string (URI) | none                    | **Required.** Full URL to scrape. Must include `http://` or `https://`.                                                                                                                                                              |
| `includeLinks`        | boolean      | `true`                  | Preserve hyperlinks in the Markdown output.                                                                                                                                                                                          |
| `includeImages`       | boolean      | `false`                 | Include image references in the Markdown output.                                                                                                                                                                                     |
| `shortenBase64Images` | boolean      | `true`                  | Truncate base64-encoded image data so it doesn't dominate the response.                                                                                                                                                              |
| `useMainContentOnly`  | boolean      | `false`                 | Strip headers, footers, sidebars, and navigation, keeping only the main content.                                                                                                                                                     |
| `includeFrames`       | boolean      | `false`                 | When true, the contents of iframes are rendered to Markdown.                                                                                                                                                                         |
| `includeSelectors`    | string\[]    | none                    | CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before conversion to Markdown. Examples: `article.main`, `#content`, `[role=main]`.                                                       |
| `excludeSelectors`    | string\[]    | none                    | CSS selectors to remove before conversion to Markdown. Applied after `includeSelectors`; exclusion takes precedence. Examples: `nav`, `footer`, `.ad-banner`.                                                                        |
| `pdf`                 | object       | `{ shouldParse: true }` | PDF-page controls: `shouldParse`, `start`, `end` (1-based inclusive range), `ocr`. Set `shouldParse: false` to skip PDFs. Set `ocr: true` to OCR images embedded in the PDF and inline the recognized text alongside the text layer. |
| `maxAgeMs`            | integer      | `86400000` (24h)        | Return a cached scrape if one exists younger than this. `0` forces a fresh scrape. Max is 30 days.                                                                                                                                   |
| `waitForMs`           | integer      | none                    | Browser wait time after initial load (max 30000). Use when the page needs JS time to populate.                                                                                                                                       |
| `headers`             | object       | none                    | Outbound HTTP headers forwarded to the target URL, sent as deep-object query params (e.g. `headers[X-Custom]=value`). When provided, caching is bypassed entirely.                                                                   |
| `timeoutMS`           | integer      | none                    | Abort with a 408 if the request exceeds this many milliseconds. Min `1000`, max `300000` (5 min).                                                                                                                                    |

### Response

```json theme={null}
{
  "success": true,
  "url": "https://example.com",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "contentLength": 174
}
```

| Field           | Type    | Description                                                                                                                                                             |
| --------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `success`       | boolean | `true` when the scrape completed.                                                                                                                                       |
| `url`           | string  | The URL that was scraped.                                                                                                                                               |
| `markdown`      | string  | The page rendered as GitHub Flavored Markdown. By default the full page is converted; pass `useMainContentOnly: true` to strip nav, footer, sidebars, and other chrome. |
| `contentLength` | integer | UTF-8 byte length of `markdown`. Use this to budget tokens or detect empty results without re-measuring the string.                                                     |

<Accordion title="Get raw HTML instead">
  To get the page as raw HTML:

  <CodeGroup>
    ```bash cURL theme={null}
    curl -G https://api.context.dev/v1/web/scrape/html \
      -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
      --data-urlencode "url=https://example.com"
    ```

    ```typescript TypeScript theme={null}
    import ContextDev from "context.dev";

    const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

    const response = await client.web.webScrapeHTML({ url: "https://example.com" });

    console.log(response.html.length);
    ```

    ```python Python theme={null}
    # Requires beautifulsoup4 and lxml if you copy this parsing example.
    import os
    from context.dev import ContextDev
    from bs4 import BeautifulSoup

    client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

    response = client.web.web_scrape_html(url="https://example.com")
    soup = BeautifulSoup(response.html, "lxml")

    print(soup.title.string)
    ```

    ```ruby Ruby theme={null}
    require "context_dev"

    client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

    response = client.web.web_scrape_html(url: "https://example.com")

    puts response.html.length
    ```

    ```go Go theme={null}
    package main

    import (
        "context"
        "fmt"
        "os"

        contextdev "github.com/context-dot-dev/context-go-sdk"
        "github.com/context-dot-dev/context-go-sdk/option"
    )

    func main() {
        client := contextdev.NewClient(
            option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
        )

        response, err := client.Web.WebScrapeHTML(context.TODO(), contextdev.WebWebScrapeHTMLParams{
            URL: "https://example.com",
        })
        if err != nil {
            panic(err)
        }

        fmt.Println(len(response.HTML))
    }
    ```

    ```php PHP theme={null}
    <?php

    use ContextDev\Client;

    $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

    $response = $client->web->webScrapeHTML(url: 'https://example.com');

    echo strlen($response->html), PHP_EOL;
    ```
  </CodeGroup>

  <Badge color="blue" icon="coins">1 credit per call</Badge>

  **Request Parameters**

  | Parameter            | Type         | Default                 | Description                                                                                                             |
  | -------------------- | ------------ | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- |
  | `url`                | string (URI) | none                    | **Required.** Full URL to scrape.                                                                                       |
  | `includeFrames`      | boolean      | `false`                 | When true, iframes are rendered inline into the returned HTML.                                                          |
  | `useMainContentOnly` | boolean      | `false`                 | Return only the page's main content, excluding headers, footers, sidebars, and navigation when detectable.              |
  | `includeSelectors`   | string\[]    | none                    | CSS selectors. When provided, only matching subtrees (and their descendants) are kept; everything else is dropped.      |
  | `excludeSelectors`   | string\[]    | none                    | CSS selectors to remove from the result. Applied after `includeSelectors`; exclusion takes precedence.                  |
  | `pdf`                | object       | `{ shouldParse: true }` | PDF-page controls; same shape as `/web/scrape/markdown`.                                                                |
  | `maxAgeMs`           | integer      | `86400000`              | Cache TTL. `0` for fresh. Max 30 days.                                                                                  |
  | `waitForMs`          | integer      | none                    | Wait after initial load (max 30000).                                                                                    |
  | `headers`            | object       | none                    | Outbound HTTP headers forwarded to the target URL (e.g. `headers[X-Custom]=value`). When provided, caching is bypassed. |
  | `timeoutMS`          | integer      | none                    | Abort with a 408 if the request exceeds this many milliseconds. Min `1000`, max `300000` (5 min).                       |

  **Response**

  ```json theme={null}
  {
    "success": true,
    "url": "https://example.com",
    "html": "<!DOCTYPE html><html lang=\"en\"><head><title>Example Domain</title><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><style>body{background:#eee;width:60vw;margin:15vh auto;font-family:system-ui,sans-serif}…</style></head><body>…</body></html>",
    "type": "html"
  }
  ```

  | Field     | Type    | Description                                                                                                                                                                                                                                                                                                                                                                                                                    |
  | --------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
  | `success` | boolean | `true` when the scrape completed.                                                                                                                                                                                                                                                                                                                                                                                              |
  | `url`     | string  | The URL that was scraped.                                                                                                                                                                                                                                                                                                                                                                                                      |
  | `html`    | string  | Rendered HTML for normal pages. For sitemaps and feeds behind an XSL stylesheet, the underlying XML. For Excel workbooks, the extracted sheets as HTML `<table>` elements (one per sheet, headed by an `<h2>`). For PowerPoint decks, the extracted slides as HTML (each slide headed by `<h2>Slide N: Title</h2>`, followed by paragraphs, `<table>` elements, and an `<h3>Notes</h3>` block when speaker notes are present). |
  | `type`    | enum    | Detected content type of `html`. One of `html`, `xml`, `json`, `text`, `csv`, `markdown`, `svg`, `pdf`, `docx`, `doc`, `xlsx`, `xls`, `pptx`, `ppt`.                                                                                                                                                                                                                                                                           |
</Accordion>

## Crawl a whole site

`POST /web/crawl` takes a seed URL and returns an array of scraped pages in one call. That's exactly the shape you want for seeding a RAG index or building a knowledge base.

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.context.dev/v1/web/crawl \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://context.dev",
      "maxPages": 3,
      "maxDepth": 1
    }'
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.web.webCrawlMd({
    url: "https://context.dev",
    maxPages: 3,
    maxDepth: 1,
  });

  for (const page of response.results) {
    console.log(`${page.metadata.url}: ${page.markdown.length} chars`);
  }
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.web.web_crawl_md(
      url="https://context.dev",
      max_pages=3,
      max_depth=1,
  )

  for page in response.results:
      print(f"{page.metadata.url}: {len(page.markdown)} chars")
  ```

  ```ruby Ruby theme={null}
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.web.web_crawl_md(
    url: "https://context.dev",
    max_pages: 3,
    max_depth: 1,
  )

  response.results.each do |page|
    puts "#{page.metadata.url}: #{page.markdown.length} chars"
  end
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      contextdev "github.com/context-dot-dev/context-go-sdk"
      "github.com/context-dot-dev/context-go-sdk/option"
      "github.com/context-dot-dev/context-go-sdk/packages/param"
  )

  func main() {
      client := contextdev.NewClient(
          option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
      )

      response, err := client.Web.WebCrawlMd(context.TODO(), contextdev.WebWebCrawlMdParams{
          URL:      "https://context.dev",
          MaxPages: param.NewOpt(int64(3)),
          MaxDepth: param.NewOpt(int64(1)),
      })
      if err != nil {
          panic(err)
      }

      for _, page := range response.Results {
          fmt.Printf("%s: %d chars\n", page.Metadata.URL, len(page.Markdown))
      }
  }
  ```

  ```php PHP theme={null}
  <?php

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->web->webCrawlMd(
      url: 'https://context.dev',
      maxPages: 3,
      maxDepth: 1,
  );

  foreach ($response->results as $page) {
      echo $page->metadata->url, ': ', strlen($page->markdown), ' chars', PHP_EOL;
  }
  ```
</CodeGroup>

<Badge color="blue" icon="coins">1 credit per page scraped</Badge>

<Warning>
  Crawls are billed per page, not per call. Scraping a website with 50 pages costs 50 credits. Set `maxPages` accordingly.
</Warning>

### Request Parameters

| Parameter             | Type         | Default                 | Description                                                                                                                           |
| --------------------- | ------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                 | string (URI) | none                    | **Required.** Starting URL for the crawl.                                                                                             |
| `maxPages`            | integer      | `100`                   | Maximum pages to crawl. Hard cap: `500`.                                                                                              |
| `maxDepth`            | integer      | none                    | Maximum link depth from the starting URL (`0` = only the seed).                                                                       |
| `urlRegex`            | string       | none                    | Only URLs matching this regex are followed and scraped. Example: `^https?://[^/]+/blog/`.                                             |
| `followSubdomains`    | boolean      | `false`                 | When true, follow links on subdomains (`docs.example.com` from `example.com`). `www` and apex are always treated as equivalent.       |
| `includeLinks`        | boolean      | `true`                  | Preserve hyperlinks in each page's Markdown.                                                                                          |
| `includeImages`       | boolean      | `false`                 | Include image references in each page's Markdown.                                                                                     |
| `shortenBase64Images` | boolean      | `true`                  | Truncate base64 image data.                                                                                                           |
| `useMainContentOnly`  | boolean      | `false`                 | Strip nav/footer/sidebars on every page.                                                                                              |
| `includeFrames`       | boolean      | `false`                 | Render iframes on every page.                                                                                                         |
| `includeSelectors`    | string\[]    | none                    | CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each page is converted to Markdown. |
| `excludeSelectors`    | string\[]    | none                    | CSS selectors to remove before each page is converted to Markdown. Applied after `includeSelectors`; exclusion takes precedence.      |
| `pdf`                 | object       | `{ shouldParse: true }` | PDF-page controls. Set `shouldParse: false` to skip PDFs entirely.                                                                    |
| `maxAgeMs`            | integer      | `86400000`              | Per-page cache TTL.                                                                                                                   |
| `waitForMs`           | integer      | none                    | Per-page wait after initial load. Max 30000.                                                                                          |
| `stopAfterMs`         | integer      | `80000`                 | Soft time budget for the entire crawl (10000–110000). The crawler returns what it has so far when exceeded.                           |
| `timeoutMS`           | integer      | none                    | Hard abort: returns a 408 if the request exceeds this many milliseconds. Min `1000`, max `300000` (5 min).                            |

### Response

```json expandable theme={null}
{
  "results": [
    {
      "markdown": "# Context.dev\n\nTurn any domain into structured, AI-ready data…",
      "metadata": {
        "url": "https://context.dev/",
        "title": "Context.dev: Brand & Web APIs for Agents",
        "crawlDepth": 0,
        "statusCode": 200,
        "success": true
      }
    },
    {
      "markdown": "# Brand.dev is now Context.dev\n\nWhy we renamed…",
      "metadata": {
        "url": "https://context.dev/blog/brand-dev-is-now-context-dev",
        "title": "Brand.dev is now Context.dev",
        "crawlDepth": 1,
        "statusCode": 200,
        "success": true
      }
    },
    {
      "markdown": "# Pricing\n\nPay only for successful calls…",
      "metadata": {
        "url": "https://context.dev/pricing",
        "title": "Pricing | Context.dev",
        "crawlDepth": 1,
        "statusCode": 200,
        "success": true
      }
    }
  ],
  "metadata": {
    "numUrls": 3,
    "maxCrawlDepth": 1,
    "numSucceeded": 3,
    "numFailed": 0,
    "numSkipped": 0
  }
}
```

| Field                           | Type    | Description                                                          |
| ------------------------------- | ------- | -------------------------------------------------------------------- |
| `results[]`                     | array   | One entry per crawled page.                                          |
| `results[].markdown`            | string  | The page body as GitHub-Flavored Markdown.                           |
| `results[].metadata.url`        | string  | The URL that was fetched (after redirects).                          |
| `results[].metadata.title`      | string  | The page's `<title>` tag value.                                      |
| `results[].metadata.crawlDepth` | number  | Link-hops from the seed URL (`0` for the seed itself).               |
| `results[].metadata.statusCode` | number  | HTTP status of the underlying fetch.                                 |
| `results[].metadata.success`    | boolean | `false` for pages that failed to render; `markdown` may be empty.    |
| `metadata.numUrls`              | number  | Total URLs the crawler attempted.                                    |
| `metadata.maxCrawlDepth`        | number  | Deepest hop reached during the crawl.                                |
| `metadata.numSucceeded`         | number  | Pages fetched successfully. Matches the credit cost.                 |
| `metadata.numFailed`            | number  | Pages that errored.                                                  |
| `metadata.numSkipped`           | number  | Pages skipped (e.g. by `urlRegex` or `pdf: { shouldParse: false }`). |

## Get all URLs of a domain

`GET /web/scrape/sitemap` reads `sitemap.xml` from a domain root, follows any nested sitemap indexes, and returns a de-duplicated URL list without rendering any of the pages. Use it for cheap coverage of large sites or to feed a downstream scraper with a curated list.

<CodeGroup>
  ```bash cURL theme={null}
  curl -G https://api.context.dev/v1/web/scrape/sitemap \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --data-urlencode "domain=stripe.com" \
    --data-urlencode "maxLinks=50" \
    --data-urlencode "urlRegex=/customers/"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.web.webScrapeSitemap({
    domain: "stripe.com",
    maxLinks: 50,
    urlRegex: "/customers/",
  });

  console.log(`${response.urls.length} URLs across ${response.meta.sitemapsDiscovered} sitemaps`);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.web.web_scrape_sitemap(
      domain="stripe.com",
      max_links=50,
      url_regex="/customers/",
  )

  print(f"{len(response.urls)} URLs across {response.meta.sitemaps_discovered} sitemaps")
  ```

  ```ruby Ruby theme={null}
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.web.web_scrape_sitemap(
    domain: "stripe.com",
    max_links: 50,
    url_regex: "/customers/",
  )

  puts "#{response.urls.length} URLs across #{response.meta.sitemaps_discovered} sitemaps"
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      contextdev "github.com/context-dot-dev/context-go-sdk"
      "github.com/context-dot-dev/context-go-sdk/option"
      "github.com/context-dot-dev/context-go-sdk/packages/param"
  )

  func main() {
      client := contextdev.NewClient(
          option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
      )

      response, err := client.Web.WebScrapeSitemap(context.TODO(), contextdev.WebWebScrapeSitemapParams{
          Domain:   "stripe.com",
          MaxLinks: param.NewOpt(int64(50)),
          URLRegex: param.NewOpt("/customers/"),
      })
      if err != nil {
          panic(err)
      }

      fmt.Printf("%d URLs across %d sitemaps\n", len(response.URLs), response.Meta.SitemapsDiscovered)
  }
  ```

  ```php PHP theme={null}
  <?php

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->web->webScrapeSitemap(
      domain: 'stripe.com',
      maxLinks: 50,
      urlRegex: '/customers/',
  );

  echo count($response->urls), ' URLs across ', $response->meta->sitemapsDiscovered, ' sitemaps', PHP_EOL;
  ```
</CodeGroup>

<Badge color="blue" icon="coins">1 credit per call</Badge>

### Request Parameters

| Parameter   | Type    | Default | Description                                                                                                                                   |
| ----------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `domain`    | string  | none    | **Required.** Domain to build a sitemap for (e.g. `example.com`). No protocol required; the API validates and normalizes the input.           |
| `maxLinks`  | integer | `500`   | Maximum URLs to return (effective range 1–500). The response's `urls[]` array is hard-capped at 500 entries, so values above 500 are clamped. |
| `urlRegex`  | string  | none    | Filter the discovered URLs by regex pattern.                                                                                                  |
| `headers`   | object  | none    | Outbound HTTP headers forwarded to the target URL (e.g. `headers[X-Custom]=value`). When provided, caching is bypassed.                       |
| `timeoutMS` | integer | none    | Abort with a 408 if the request exceeds this many milliseconds. Min `1000`, max `300000` (5 min).                                             |

### Response

```json theme={null}
{
  "success": true,
  "domain": "stripe.com",
  "urls": [
    "https://stripe.com/customers/all",
    "https://stripe.com/customers/gamma",
    "https://stripe.com/customers/chatbase"
  ],
  "meta": {
    "sitemapsDiscovered": 4,
    "sitemapsFetched": 4,
    "sitemapsSkipped": 0,
    "errors": 0
  }
}
```

| Field                     | Type      | Description                                                                                        |
| ------------------------- | --------- | -------------------------------------------------------------------------------------------------- |
| `success`                 | boolean   | `true` when the sitemap crawl completed.                                                           |
| `domain`                  | string    | The normalized domain that was crawled.                                                            |
| `urls[]`                  | string\[] | Discovered page URLs, de-duplicated. Bounded by `maxLinks` and capped at 500 entries per response. |
| `meta.sitemapsDiscovered` | number    | Total sitemap XML files discovered (root + nested indexes).                                        |
| `meta.sitemapsFetched`    | number    | Sitemaps actually fetched and parsed.                                                              |
| `meta.sitemapsSkipped`    | number    | Sitemaps skipped (404s, malformed XML, etc.).                                                      |
| `meta.errors`             | number    | Errors encountered during crawling.                                                                |

## Extract every image on a page

`GET /web/scrape/images` takes a URL and returns a manifest of every image referenced on the page: `<img>` tags, inline `<svg>`, CSS background images, `<picture>` sources, OpenGraph and Twitter card images, favicons.

Opt into `enrichment` to also get measured dimensions, a CDN-hosted copy, and a visual-type classification per image.

<CodeGroup>
  ```bash cURL theme={null}
  curl -G https://api.context.dev/v1/web/scrape/images \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --data-urlencode "url=https://airbnb.com" \
    --data-urlencode "enrichment[resolution]=true" \
    --data-urlencode "enrichment[classification]=true"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.web.webScrapeImages({
    url: "https://airbnb.com",
    enrichment: { resolution: true, classification: true },
  });

  const hero = response.images
    .filter((img) => img.enrichment?.width && img.enrichment?.height)
    .sort((a, b) => b.enrichment!.width! * b.enrichment!.height! - a.enrichment!.width! * a.enrichment!.height!)[0];

  console.log(hero?.src, hero?.enrichment?.type);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.web.web_scrape_images(
      url="https://airbnb.com",
      enrichment={"resolution": True, "classification": True},
  )

  sized = [img for img in response.images if img.enrichment and img.enrichment.width and img.enrichment.height]
  hero = max(sized, key=lambda i: i.enrichment.width * i.enrichment.height, default=None)
  if hero:
      print(hero.src, hero.enrichment.type)
  ```

  ```ruby Ruby theme={null}
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.web.web_scrape_images(
    url: "https://airbnb.com",
    enrichment: { resolution: true, classification: true },
  )

  sized = response.images.select { |i| i.enrichment && i.enrichment.width && i.enrichment.height }
  hero = sized.max_by { |i| i.enrichment.width * i.enrichment.height }
  puts "#{hero.src} #{hero.enrichment.type}" if hero
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      contextdev "github.com/context-dot-dev/context-go-sdk"
      "github.com/context-dot-dev/context-go-sdk/option"
      "github.com/context-dot-dev/context-go-sdk/packages/param"
  )

  func main() {
      client := contextdev.NewClient(
          option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
      )

      response, err := client.Web.WebScrapeImages(context.TODO(), contextdev.WebWebScrapeImagesParams{
          URL: "https://airbnb.com",
          Enrichment: contextdev.WebWebScrapeImagesParamsEnrichment{
              Resolution:     param.NewOpt(true),
              Classification: param.NewOpt(true),
          },
      })
      if err != nil {
          panic(err)
      }

      fmt.Printf("found %d images\n", len(response.Images))
  }
  ```

  ```php PHP theme={null}
  <?php

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->web->webScrapeImages(
      url: 'https://airbnb.com',
      enrichment: ['resolution' => true, 'classification' => true],
  );

  $sized = array_filter(
      $response->images ?? [],
      fn ($img) => $img->enrichment?->width && $img->enrichment?->height,
  );

  usort($sorted = array_values($sized), fn ($a, $b) =>
      ($b->enrichment->width * $b->enrichment->height) <=> ($a->enrichment->width * $a->enrichment->height)
  );
  $hero = $sorted[0] ?? null;

  if ($hero) {
      echo $hero->src, ' ', $hero->enrichment->type, PHP_EOL;
  }
  ```
</CodeGroup>

<Badge color="blue" icon="coins">1 credit per call</Badge>
<Badge color="blue" icon="coins">5 credits per call if enrichment flags are used</Badge>

### Request Parameters

| Parameter                   | Type         | Default    | Description                                                                                                             |
| --------------------------- | ------------ | ---------- | ----------------------------------------------------------------------------------------------------------------------- |
| `url`                       | string (URI) | none       | **Required.** Page URL to inspect.                                                                                      |
| `maxAgeMs`                  | integer      | `86400000` | Cache TTL (0 forces fresh; max 30 days).                                                                                |
| `enrichment.resolution`     | boolean      | `false`    | Measure width × height in pixels when possible.                                                                         |
| `enrichment.hostedUrl`      | boolean      | `false`    | Host materializable images on Context.dev's CDN and return their URL + MIME type.                                       |
| `enrichment.classification` | boolean      | `false`    | Classify each image as `photography`, `illustration`, `logo`, `wordmark`, `icon`, `pattern`, `graphic`, or `other`.     |
| `enrichment.maxTimePerMs`   | integer      | `30000`    | Per-image enrichment timeout (1–60000 ms).                                                                              |
| `waitForMs`                 | integer      | none       | Browser wait after initial load (max 30000).                                                                            |
| `headers`                   | object       | none       | Outbound HTTP headers forwarded to the target URL (e.g. `headers[X-Custom]=value`). When provided, caching is bypassed. |
| `timeoutMS`                 | integer      | none       | Abort with a 408 if the request exceeds this many milliseconds. Min `1000`, max `300000` (5 min).                       |

### Response

```json expandable theme={null}
{
  "success": true,
  "url": "https://airbnb.com",
  "images": [
    {
      "src": "https://a0.muscache.com/im/pictures/airbnb-platform-assets/AirbnbPlatformAssets-UserProfile/original/5347d650-16de-4f5a-a38e-79edc988befa.png?im_w=720",
      "element": "img",
      "type": "url",
      "alt": null,
      "enrichment": {
        "width": 720,
        "height": 720,
        "type": "illustration"
      }
    },
    {
      "src": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 32 32\" …></svg>",
      "element": "svg",
      "type": "html",
      "alt": null,
      "enrichment": {
        "width": 16,
        "height": 16,
        "type": "icon"
      }
    }
  ]
}
```

| Field                          | Type           | Description                                                                                                                                                      |
| ------------------------------ | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `success`                      | boolean        | `true` when the scrape completed.                                                                                                                                |
| `url`                          | string         | The page URL that was scraped.                                                                                                                                   |
| `images[]`                     | array          | One entry per image referenced on the page.                                                                                                                      |
| `images[].src`                 | string         | For `type: "url"`, the absolute image URL. For `type: "html"`, the raw inline SVG/HTML.                                                                          |
| `images[].element`             | enum           | DOM origin: `img`, `svg`, `link`, `source`, `video`, `css`, `object`, `meta`, or `background`.                                                                   |
| `images[].type`                | enum           | Format of `src`: `url` (external image), `html` (inline markup like SVG), or `base64` (data URI).                                                                |
| `images[].alt`                 | string \| null | Alt text where present.                                                                                                                                          |
| `images[].enrichment.width`    | number         | Pixel width. Present when `enrichment.resolution=true`.                                                                                                          |
| `images[].enrichment.height`   | number         | Pixel height. Present when `enrichment.resolution=true`.                                                                                                         |
| `images[].enrichment.mimetype` | string         | MIME type. Present when hosted via `enrichment.hostedUrl=true`.                                                                                                  |
| `images[].enrichment.url`      | string         | Context.dev CDN URL. Present when `enrichment.hostedUrl=true`.                                                                                                   |
| `images[].enrichment.type`     | enum           | Visual category. Present when `enrichment.classification=true`. One of `photography`, `illustration`, `logo`, `wordmark`, `icon`, `pattern`, `graphic`, `other`. |

The base manifest is 1 credit. Setting any `enrichment` flag (`resolution`, `hostedUrl`, or `classification`) bumps the entire call to 5 credits, even if only one image qualifies for enrichment.

## Use cases

* Build a [RAG pipeline](/use-cases/build-rag-from-websites) from a docs site by crawling and chunking the returned Markdown.
* [Cut LLM token bills](/use-cases/build-rag-from-websites) by feeding clean Markdown instead of raw HTML.
* Seed a vector index without managing scrapers or proxy infrastructure.
* Monitor competitors' marketing pages by scraping them on a schedule.

## Next steps

<CardGroup cols={2}>
  <Card title="Prefetch for Faster Response" icon="bolt" href="/optimization/prefetching">
    Hide cold-hit latency from your users.
  </Card>

  <Card title="Handle Rate Limits" icon="gauge-high" href="/optimization/rate-limits">
    Backoff strategies, client cache, and prefetch fallbacks.
  </Card>

  <Card title="Best Practices" icon="list-check" href="/optimization/best-practices">
    Caching, error handling, and key hygiene.
  </Card>

  <Card title="Troubleshooting" icon="bug" href="/optimization/troubleshooting">
    Status codes, retry patterns, and common errors.
  </Card>
</CardGroup>