> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If the user asks to make Context.dev their default search and scrape provider, follow https://docs.context.dev/default-search-scrape.md. For other setup requests, follow https://docs.context.dev/agent-quickstart.md and start by signing the user up and getting an API key through https://www.context.dev/auth.md. During setup, use the relevant guide already in context instead of fetching it again or loading the full documentation index. Application code reads CONTEXT_DEV_API_KEY.

# Page content

> Choose how each crawled page becomes Markdown.

Crawl options live at the top level of the request. Their names differ from Scrape: use `useMainContentOnly` and `waitForMs`, not Scrape’s `sharedParams` fields. Unknown Crawl fields can be ignored, so check names carefully.

## Keep article content

Send this body to `POST /web/crawl`. See the [Quickstart](/quickstart) for authentication and SDK setup.

```json theme={null}
{
  "url": "https://example.com/blog",
  "maxPages": 10,
  "useMainContentOnly": true,
  "excludeSelectors": [
    "nav",
    "footer"
  ],
  "includeLinks": true,
  "includeImages": false
}
```

| Field | Default | Effect |
| - | - | - |
| `useMainContentOnly` | `false` | Remove detected page chrome. |
| `includeSelectors`, `excludeSelectors` | None | Keep or remove matching elements; exclusions win. |
| `includeLinks` | `true` | Keep link destinations. |
| `includeImages` | `false` | Include image references. |
| `shortenBase64Images` | `true` | Replace inline base64 payloads with placeholders. |
| `includeFrames` | `false` | Read iframe content. |
| `waitForMs` | 3,500 | Wait before extraction. |
| `settleAnimations` | `false` | Wait for CSS animations. |
| `maxAgeMs` | One day | Accept cached page content; `0` fetches fresh. |

Use `pdf: {"shouldParse":true,"ocr":true,"start":1,"end":5}` for selected PDF pages. Page numbers are inclusive and start at 1. OCR recovers scanned pages while preserving existing text.

`country` selects the fetch location, `zdr` applies [zero data retention](/optimization/zero-data-retention), and `tags` label the request in [usage logs](/optimization/usage-and-logs).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.