> ## Documentation Index > Fetch the complete documentation index at: https://docs.context.dev/llms.txt > Use this file to discover all available pages before exploring further. # Crawl Website Content > Crawl a site and return clean Markdown pages ready for RAG, search, and content pipelines. 1 Credit Per Page ## OpenAPI ````yaml POST /web/crawl openapi: 3.1.0 info: title: Context API description: API for retrieving context data from any website version: 1.0.0 servers: - url: https://api.context.dev/v1 security: [] tags: - name: Monitors description: >- Monitor pages, sitemaps, and extracted website data for exact or semantic changes. Webhook payloads are documented by the MonitorsChangeDetectedWebhookPayload and MonitorsRunCompletedWebhookPayload schemas. paths: /web/crawl: post: tags: - Web Scraping summary: Crawl Website & Scrape Markdown description: >- Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages. requestBody: required: true content: application/json: schema: type: object properties: url: type: string format: uri description: >- The starting URL for the crawl (must include http:// or https:// protocol) maxPages: type: integer minimum: 1 maximum: 500 default: 100 description: 'Maximum number of pages to crawl. Hard cap: 500.' maxDepth: type: integer minimum: 0 description: >- Maximum link depth from the starting URL (0 = only the starting page) urlRegex: type: string description: >- Regex pattern. Only URLs matching this pattern will be followed and scraped. example: ^https?://[^/]+/blog/ includeLinks: type: boolean default: true description: Preserve hyperlinks in the Markdown output includeImages: type: boolean default: false description: Include image references in the Markdown output shortenBase64Images: type: boolean default: true description: Truncate base64-encoded image data in the Markdown output useMainContentOnly: type: boolean default: false description: >- Extract only the main content, stripping headers, footers, sidebars, and navigation followSubdomains: type: boolean default: false description: >- When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent. pdf: type: object properties: shouldParse: type: boolean default: true description: >- When true, PDF pages are fetched and parsed. When false, PDF pages are skipped entirely (not included in results and not counted as failures). ocr: type: boolean default: false description: >- When true, detect and OCR images embedded in the selected PDF pages, inserting recognized text at each image's position in page reading order while preserving the PDF text layer. This is separate from automatic scanned-PDF OCR fallback. start: type: integer minimum: 1 description: >- First 1-based PDF page to parse. When omitted, parsing starts at the first page. end: type: integer minimum: 1 description: >- Last 1-based PDF page to parse. When omitted, parsing ends at the last page. Must be greater than or equal to start when both are provided. additionalProperties: false default: shouldParse: true ocr: false description: >- PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. includeFrames: type: boolean default: false description: >- When true, the contents of iframes are rendered to Markdown for each crawled page. includeSelectors: type: array maxItems: 50 items: type: string maxLength: 2048 description: >- CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each crawled page is converted to Markdown. When omitted, the entire document is kept. Examples: "article.main", "#content", "[role=main]". excludeSelectors: type: array maxItems: 50 items: type: string maxLength: 2048 description: >- CSS selectors to remove before each crawled page is converted to Markdown. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: "nav", "footer", ".ad-banner", "[aria-hidden=true]". maxAgeMs: type: integer minimum: 0 maximum: 2592000000 default: 86400000 description: >- Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. waitForMs: type: integer minimum: 0 maximum: 30000 description: >- Optional browser wait time in milliseconds after initial page load for each crawled page. Min: 0. Max: 30000 (30 seconds). settleAnimations: type: boolean default: false description: >- When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages. stopAfterMs: type: integer minimum: 10000 maximum: 110000 default: 80000 description: >- Soft time budget for the crawl in milliseconds. After each scrape, the crawler checks the elapsed time and, if exceeded, returns the pages collected so far instead of continuing. Min: 10000 (10s). Max: 110000 (110s). Default: 80000 (80s). country: $ref: '#/components/schemas/BrowserCountryCode' timeoutMS: $ref: '#/components/schemas/TimeoutMS' tags: $ref: '#/components/schemas/RequestTags' required: - url responses: '200': description: Successful response content: application/json: schema: type: object properties: results: type: array items: type: object properties: markdown: type: string description: >- Extracted page content as Markdown (empty string on failure) metadata: allOf: - $ref: '#/components/schemas/PageMetadata' - type: object properties: url: type: string description: The crawl URL fetched for this page. title: type: string description: >- Best page title extracted from the page (empty string if unavailable). crawlDepth: type: integer description: >- Depth relative to the start URL. 0 = start URL, 1 = one link away. statusCode: type: integer description: HTTP status code of the response success: type: boolean description: >- true if the page was fetched and parsed successfully required: - url - title - crawlDepth - statusCode - success required: - markdown - metadata metadata: type: object properties: numUrls: type: integer description: Total number of URLs crawled maxCrawlDepth: type: integer description: Maximum crawl depth reached during the crawl numSucceeded: type: integer description: Number of pages successfully crawled numFailed: type: integer description: Number of pages that failed to crawl numSkipped: type: integer description: >- Number of URLs skipped (PDFs when pdf.shouldParse=false, or URLs not matching urlRegex) required: - numUrls - maxCrawlDepth - numSucceeded - numFailed - numSkipped key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - results - metadata headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '400': description: Bad request - Invalid URL or parameters content: application/json: schema: type: object properties: message: type: string description: Error message describing the issue error_code: type: string enum: - INPUT_VALIDATION_ERROR - WEBSITE_ACCESS_ERROR description: Error code indicating the type of error key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '401': description: Unauthorized - Invalid or missing API key content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - UNAUTHORIZED description: Error code indicating unauthorized access key_metadata: $ref: '#/components/schemas/KeyMetadata' headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '403': description: Forbidden - Insufficient permissions or usage limit exceeded content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - FORBIDDEN - USAGE_EXCEEDED - DISABLED - INSUFFICIENT_PERMISSIONS description: Error code indicating forbidden access key_metadata: $ref: '#/components/schemas/KeyMetadata' headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '404': description: Start URL returned a 404 content: application/json: schema: type: object properties: message: type: string description: Error message describing the issue error_code: type: string enum: - NOT_FOUND description: Error code indicating the start URL was not found key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '408': description: Request timeout content: application/json: schema: type: object properties: message: type: string description: Timeout error message error_code: type: string enum: - REQUEST_TIMEOUT description: Error code indicating request timeout key_metadata: $ref: '#/components/schemas/KeyMetadata' headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '415': description: >- Unsupported content type - the start URL resolved to a content type that is not supported (e.g. an image, presentation, media, or archive). Supported types are HTML, XML, PDF, DOCX, DOC, XLSX, XLS, PPTX, PPT, and CSV. content: application/json: schema: type: object properties: message: type: string description: Error message describing the unsupported content type error_code: type: string enum: - UNSUPPORTED_CONTENT description: Error code indicating an unsupported content type key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' '429': $ref: '#/components/responses/RateLimited' '500': description: Internal server error content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - INTERNAL_ERROR description: Error code indicating internal server error key_metadata: $ref: '#/components/schemas/KeyMetadata' headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' security: - bearerAuth: [] x-codeSamples: - lang: JavaScript source: >- import ContextDev from 'context.dev'; const client = new ContextDev({ apiKey: process.env['CONTEXT_DEV_API_KEY'], // This is the default and can be omitted }); const response = await client.web.webCrawlMd({ url: 'https://example.com' }); console.log(response.metadata); - lang: Python source: |- import os from context.dev import ContextDev client = ContextDev( api_key=os.environ.get("CONTEXT_DEV_API_KEY"), # This is the default and can be omitted ) response = client.web.web_crawl_md( url="https://example.com", ) print(response.metadata) - lang: Go source: "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/context-dot-dev/context-go-sdk\"\n\t\"github.com/context-dot-dev/context-go-sdk/option\"\n)\n\nfunc main() {\n\tclient := contextdev.NewClient(\n\t\toption.WithAPIKey(\"My API Key\"),\n\t)\n\tresponse, err := client.Web.WebCrawlMd(context.TODO(), contextdev.WebWebCrawlMdParams{\n\t\tURL: \"https://example.com\",\n\t})\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tfmt.Printf(\"%+v\\n\", response.Metadata)\n}\n" - lang: Ruby source: |- require "context_dev" context_dev = ContextDev::Client.new(api_key: "My API Key") response = context_dev.web.web_crawl_md(url: "https://example.com") puts(response) - lang: PHP source: >- web->webCrawlMd( url: 'https://example.com', country: 'de', excludeSelectors: ['string'], followSubdomains: true, includeFrames: true, includeImages: true, includeLinks: true, includeSelectors: ['string'], maxAgeMs: 0, maxDepth: 0, maxPages: 1, pdf: ['end' => 1, 'ocr' => true, 'shouldParse' => true, 'start' => 1], settleAnimations: true, shortenBase64Images: true, stopAfterMs: 10000, tags: ['production', 'team-alpha'], timeoutMs: 1000, urlRegex: '^https?://[^/]+/blog/', useMainContentOnly: true, waitForMs: 0, ); var_dump($response); } catch (APIException $e) { echo $e->getMessage(); } - lang: CLI source: |- context-dev web web-crawl-md \ --api-key 'My API Key' \ --url https://example.com components: schemas: BrowserCountryCode: type: string enum: - ad - ae - af - ag - ai - al - am - ao - ar - at - au - aw - az - ba - bb - bd - be - bf - bg - bh - bi - bj - bm - bn - bo - bq - br - bs - bw - by - bz - ca - cd - cf - cg - ch - ci - cl - cm - cn - co - cr - cv - cw - cy - cz - de - dj - dk - dm - do - dz - ec - ee - eg - es - et - fi - fj - fr - ga - gb - gd - ge - gf - gg - gh - gm - gn - gp - gq - gr - gt - gu - gw - gy - hk - hn - hr - ht - hu - id - ie - il - im - in - iq - ir - is - it - je - jm - jo - jp - ke - kg - kh - kn - kr - kw - ky - kz - la - lb - lc - lk - lr - ls - lt - lu - lv - ly - ma - mc - md - me - mf - mg - mk - ml - mm - mn - mo - mq - mr - mt - mu - mv - mw - mx - my - mz - na - nc - ne - ng - ni - nl - 'no' - np - nz - om - pa - pe - pf - pg - ph - pk - pl - pr - ps - pt - py - qa - re - ro - rs - ru - rw - sa - sc - sd - se - sg - si - sk - sl - sm - sn - so - sr - ss - st - sv - sx - sy - sz - tc - td - tg - th - tj - tl - tm - tn - tr - tt - tw - tz - ua - ug - us - uy - uz - vc - ve - vg - vi - vn - ye - yt - za - zm - zw example: de description: >- Two-letter ISO 3166-1 alpha-2 country code identifying a supported Context.dev residential proxy exit location. Must be one of Context.dev's supported countries. When provided, Context.dev fetches the target page from that country. TimeoutMS: type: integer minimum: 1000 maximum: 300000 description: >- Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes). RequestTags: type: array items: type: string minLength: 1 maxLength: 50 maxItems: 20 description: >- Optional caller-defined tags for tracking this request. Tags are recorded on the request's usage log and can be used to filter usage on the dashboard usage page. Up to 20 tags, each 1-50 characters. example: - production - team-alpha PageMetadata: type: object description: Metadata extracted from the scraped page HTML. properties: sourceUrl: type: string description: Original URL requested by the caller. finalUrl: type: string description: >- Final URL scraped after redirects or scraper fallback, when known. Falls back to sourceUrl when unavailable. title: type: string description: Best title extracted from the page. description: type: string description: >- Best description extracted from standard, Open Graph, or Twitter metadata. language: type: string description: Language extracted from html lang or language meta tags. keywords: type: array description: Keywords extracted from the page's keywords meta tag. items: type: string canonicalUrl: type: string description: Resolved canonical URL, when present. author: type: string description: Author metadata, when present. siteName: type: string description: Site or application name from page metadata. image: type: string description: >- Primary resolved preview image from Open Graph, Twitter, or image metadata. favicon: type: string description: Resolved favicon URL, when present. publishedTime: type: string description: Published timestamp/date from page metadata, when present. modifiedTime: type: string description: Modified timestamp/date from page metadata, when present. robots: type: string description: Robots meta directive, when present. openGraph: type: object description: >- Open Graph metadata with the og: prefix removed and keys camel-cased. additionalProperties: $ref: '#/components/schemas/PageMetadataValue' twitter: type: object description: >- Twitter card metadata with the twitter: prefix removed and keys camel-cased. additionalProperties: $ref: '#/components/schemas/PageMetadataValue' alternates: type: array description: Resolved alternate links from link rel=alternate tags. items: $ref: '#/components/schemas/PageMetadataAlternate' jsonLd: type: array description: JSON-LD structured data blocks parsed from the page. items: type: object additionalProperties: true additionalMeta: type: object description: >- Additional non-social meta tags not promoted to top-level metadata fields. additionalProperties: $ref: '#/components/schemas/PageMetadataValue' required: - sourceUrl - finalUrl KeyMetadata: type: object description: >- Metadata about the API key used for the request. Included in every response whenever a valid API key is provided, even when the response status is not 200. properties: credits_consumed: type: integer description: The number of credits consumed by this request. credits_remaining: type: integer description: >- The number of credits remaining for your organization after this request. required: - credits_consumed - credits_remaining PageMetadataValue: oneOf: - type: string - type: array items: type: string PageMetadataAlternate: type: object properties: href: type: string description: Resolved alternate URL. hreflang: type: string description: Language or locale for the alternate URL, when present. type: type: string description: Alternate resource MIME type, when present. title: type: string description: Alternate resource title, when present. required: - href additionalProperties: false headers: RateLimitLimit: description: >- Maximum requests allowed in the current fixed one-minute window. Returned when the authenticated API key has a per-minute rate limit. schema: type: integer minimum: 1 RateLimitRemaining: description: >- Requests remaining in the current fixed one-minute window. Returned when the authenticated API key has a per-minute rate limit. schema: type: integer minimum: 0 RateLimitReset: description: >- Unix timestamp in seconds when the current rate-limit window resets. Returned when the authenticated API key has a per-minute rate limit. schema: type: integer responses: RateLimited: description: Rate limit exceeded headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' Retry-After: description: Seconds until the per-minute rate limit window resets schema: type: integer minimum: 1 maximum: 60 content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - RATE_LIMITED description: Error code indicating the rate limit was exceeded key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code securitySchemes: bearerAuth: type: http scheme: bearer ````