> ## Documentation Index > Fetch the complete documentation index at: https://docs.context.dev/llms.txt > Use this file to discover all available pages before exploring further. # Crawl Website Content > Crawl a site and return clean Markdown pages ready for RAG, search, and content pipelines. 1 Credit Per Page ## OpenAPI ````yaml POST /web/crawl openapi: 3.0.0 info: title: Context API description: API for retrieving context data from any website version: 1.0.0 servers: - url: https://api.context.dev/v1 security: [] tags: - name: Monitors description: >- Monitor pages, sitemaps, and extracted website data for exact or semantic changes. The change.detected webhook payload is documented by the MonitorsChangeDetectedWebhookPayload schema. paths: /web/crawl: post: tags: - Web Scraping summary: Crawl Website & Scrape Markdown description: >- Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages. requestBody: required: true content: application/json: schema: type: object properties: url: type: string format: uri description: >- The starting URL for the crawl (must include http:// or https:// protocol) maxPages: type: integer minimum: 1 maximum: 500 default: 100 description: 'Maximum number of pages to crawl. Hard cap: 500.' maxDepth: type: integer minimum: 0 description: >- Maximum link depth from the starting URL (0 = only the starting page) urlRegex: type: string description: >- Regex pattern. Only URLs matching this pattern will be followed and scraped. example: ^https?://[^/]+/blog/ includeLinks: type: boolean default: true description: Preserve hyperlinks in the Markdown output includeImages: type: boolean default: false description: Include image references in the Markdown output shortenBase64Images: type: boolean default: true description: Truncate base64-encoded image data in the Markdown output useMainContentOnly: type: boolean default: false description: >- Extract only the main content, stripping headers, footers, sidebars, and navigation followSubdomains: type: boolean default: false description: >- When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent. pdf: type: object properties: shouldParse: type: boolean default: true description: >- When true, PDF pages are fetched and parsed. When false, PDF pages are skipped entirely (not included in results and not counted as failures). start: type: integer minimum: 1 description: >- First 1-based PDF page to parse. When omitted, parsing starts at the first page. end: type: integer minimum: 1 description: >- Last 1-based PDF page to parse. When omitted, parsing ends at the last page. Must be greater than or equal to start when both are provided. additionalProperties: false default: shouldParse: true description: >- PDF parsing controls. Use start/end to limit text extraction and OCR to an inclusive 1-based page range. includeFrames: type: boolean default: false description: >- When true, the contents of iframes are rendered to Markdown for each crawled page. includeSelectors: type: array maxItems: 50 items: type: string maxLength: 2048 description: >- CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each crawled page is converted to Markdown. When omitted, the entire document is kept. Examples: "article.main", "#content", "[role=main]". excludeSelectors: type: array maxItems: 50 items: type: string maxLength: 2048 description: >- CSS selectors to remove before each crawled page is converted to Markdown. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: "nav", "footer", ".ad-banner", "[aria-hidden=true]". maxAgeMs: type: integer minimum: 0 maximum: 2592000000 default: 86400000 description: >- Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. waitForMs: type: integer minimum: 0 maximum: 30000 description: >- Optional browser wait time in milliseconds after initial page load for each crawled page. Min: 0. Max: 30000 (30 seconds). settleAnimations: type: boolean default: false description: >- When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages. stopAfterMs: type: integer minimum: 10000 maximum: 110000 default: 80000 description: >- Soft time budget for the crawl in milliseconds. After each scrape, the crawler checks the elapsed time and, if exceeded, returns the pages collected so far instead of continuing. Min: 10000 (10s). Max: 110000 (110s). Default: 80000 (80s). country: $ref: '#/components/schemas/BrowserCountryCode' timeoutMS: $ref: '#/components/schemas/TimeoutMS' required: - url responses: '200': description: Successful response content: application/json: schema: type: object properties: results: type: array items: type: object properties: markdown: type: string description: >- Extracted page content as Markdown (empty string on failure) metadata: allOf: - $ref: '#/components/schemas/PageMetadata' - type: object properties: url: type: string description: The crawl URL fetched for this page. title: type: string description: >- Best page title extracted from the page (empty string if unavailable). crawlDepth: type: integer description: >- Depth relative to the start URL. 0 = start URL, 1 = one link away. statusCode: type: integer description: HTTP status code of the response success: type: boolean description: >- true if the page was fetched and parsed successfully required: - url - title - crawlDepth - statusCode - success required: - markdown - metadata metadata: type: object properties: numUrls: type: integer description: Total number of URLs crawled maxCrawlDepth: type: integer description: Maximum crawl depth reached during the crawl numSucceeded: type: integer description: Number of pages successfully crawled numFailed: type: integer description: Number of pages that failed to crawl numSkipped: type: integer description: >- Number of URLs skipped (PDFs when pdf.shouldParse=false, or URLs not matching urlRegex) required: - numUrls - maxCrawlDepth - numSucceeded - numFailed - numSkipped key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - results - metadata '400': description: Bad request - Invalid URL or parameters content: application/json: schema: type: object properties: message: type: string description: Error message describing the issue error_code: type: string enum: - INPUT_VALIDATION_ERROR - WEBSITE_ACCESS_ERROR description: Error code indicating the type of error key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code '401': description: Unauthorized - Invalid or missing API key content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - UNAUTHORIZED description: Error code indicating unauthorized access key_metadata: $ref: '#/components/schemas/KeyMetadata' '403': description: Forbidden - Insufficient permissions or usage limit exceeded content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - FORBIDDEN - USAGE_EXCEEDED - DISABLED - INSUFFICIENT_PERMISSIONS description: Error code indicating forbidden access key_metadata: $ref: '#/components/schemas/KeyMetadata' '404': description: Start URL returned a 404 content: application/json: schema: type: object properties: message: type: string description: Error message describing the issue error_code: type: string enum: - NOT_FOUND description: Error code indicating the start URL was not found key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code '408': description: Request timeout content: application/json: schema: type: object properties: message: type: string description: Timeout error message error_code: type: string enum: - REQUEST_TIMEOUT description: Error code indicating request timeout key_metadata: $ref: '#/components/schemas/KeyMetadata' '415': description: >- Unsupported content type - the start URL resolved to a content type that is not supported (e.g. an image, presentation, media, or archive). Supported types are HTML, XML, PDF, DOCX, DOC, XLSX, XLS, PPTX, PPT, and CSV. content: application/json: schema: type: object properties: message: type: string description: Error message describing the unsupported content type error_code: type: string enum: - UNSUPPORTED_CONTENT description: Error code indicating an unsupported content type key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code '429': $ref: '#/components/responses/RateLimited' '500': description: Internal server error content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - INTERNAL_ERROR description: Error code indicating internal server error key_metadata: $ref: '#/components/schemas/KeyMetadata' security: - bearerAuth: [] x-codeSamples: - lang: JavaScript source: >- import ContextDev from 'context.dev'; const client = new ContextDev({ apiKey: process.env['CONTEXT_DEV_API_KEY'], // This is the default and can be omitted }); const response = await client.web.webCrawlMd({ url: 'https://example.com' }); console.log(response.metadata); - lang: Python source: |- import os from context.dev import ContextDev client = ContextDev( api_key=os.environ.get("CONTEXT_DEV_API_KEY"), # This is the default and can be omitted ) response = client.web.web_crawl_md( url="https://example.com", ) print(response.metadata) - lang: Go source: "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/context-dot-dev/context-go-sdk\"\n\t\"github.com/context-dot-dev/context-go-sdk/option\"\n)\n\nfunc main() {\n\tclient := contextdev.NewClient(\n\t\toption.WithAPIKey(\"My API Key\"),\n\t)\n\tresponse, err := client.Web.WebCrawlMd(context.TODO(), contextdev.WebWebCrawlMdParams{\n\t\tURL: \"https://example.com\",\n\t})\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tfmt.Printf(\"%+v\\n\", response.Metadata)\n}\n" - lang: Ruby source: |- require "context_dev" context_dev = ContextDev::Client.new(api_key: "My API Key") response = context_dev.web.web_crawl_md(url: "https://example.com") puts(response) - lang: PHP source: >- web->webCrawlMd( url: 'https://example.com', country: 'de', excludeSelectors: ['string'], followSubdomains: true, includeFrames: true, includeImages: true, includeLinks: true, includeSelectors: ['string'], maxAgeMs: 0, maxDepth: 0, maxPages: 1, pdf: ['end' => 1, 'shouldParse' => true, 'start' => 1], settleAnimations: true, shortenBase64Images: true, stopAfterMs: 10000, timeoutMs: 1000, urlRegex: '^https?://[^/]+/blog/', useMainContentOnly: true, waitForMs: 0, ); var_dump($response); } catch (APIException $e) { echo $e->getMessage(); } - lang: CLI source: |- context-dev web web-crawl-md \ --api-key 'My API Key' \ --url https://example.com components: schemas: BrowserCountryCode: type: string enum: - ad - ae - af - ag - ai - al - am - ao - ar - at - au - aw - az - ba - bb - bd - be - bf - bg - bh - bi - bj - bm - bn - bo - bq - br - bs - bw - by - bz - ca - cd - cf - cg - ch - ci - cl - cm - cn - co - cr - cv - cw - cy - cz - de - dj - dk - dm - do - dz - ec - ee - eg - es - et - fi - fj - fr - ga - gb - gd - ge - gf - gg - gh - gm - gn - gp - gq - gr - gt - gu - gw - gy - hk - hn - hr - ht - hu - id - ie - il - im - in - iq - ir - is - it - je - jm - jo - jp - ke - kg - kh - kn - kr - kw - ky - kz - la - lb - lc - lk - lr - ls - lt - lu - lv - ly - ma - mc - md - me - mf - mg - mk - ml - mm - mn - mo - mq - mr - mt - mu - mv - mw - mx - my - mz - na - nc - ne - ng - ni - nl - 'no' - np - nz - om - pa - pe - pf - pg - ph - pk - pl - pr - ps - pt - py - qa - re - ro - rs - ru - rw - sa - sc - sd - se - sg - si - sk - sl - sm - sn - so - sr - ss - st - sv - sx - sy - sz - tc - td - tg - th - tj - tl - tm - tn - tr - tt - tw - tz - ua - ug - us - uy - uz - vc - ve - vg - vi - vn - ye - yt - za - zm - zw example: de description: >- Two-letter ISO 3166-1 alpha-2 country code identifying a supported Context.dev residential proxy exit location. Must be one of Context.dev's supported countries. When provided, Context.dev fetches the target page from that country. TimeoutMS: type: integer minimum: 1000 maximum: 300000 description: >- Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes). PageMetadata: type: object description: Metadata extracted from the scraped page HTML. properties: sourceUrl: type: string description: Original URL requested by the caller. finalUrl: type: string description: >- Final URL scraped after redirects or scraper fallback, when known. Falls back to sourceUrl when unavailable. title: type: string description: Best title extracted from the page. description: type: string description: >- Best description extracted from standard, Open Graph, or Twitter metadata. language: type: string description: Language extracted from html lang or language meta tags. keywords: type: array description: Keywords extracted from the page's keywords meta tag. items: type: string canonicalUrl: type: string description: Resolved canonical URL, when present. author: type: string description: Author metadata, when present. siteName: type: string description: Site or application name from page metadata. image: type: string description: >- Primary resolved preview image from Open Graph, Twitter, or image metadata. favicon: type: string description: Resolved favicon URL, when present. publishedTime: type: string description: Published timestamp/date from page metadata, when present. modifiedTime: type: string description: Modified timestamp/date from page metadata, when present. robots: type: string description: Robots meta directive, when present. openGraph: type: object description: >- Open Graph metadata with the og: prefix removed and keys camel-cased. additionalProperties: $ref: '#/components/schemas/PageMetadataValue' twitter: type: object description: >- Twitter card metadata with the twitter: prefix removed and keys camel-cased. additionalProperties: $ref: '#/components/schemas/PageMetadataValue' alternates: type: array description: Resolved alternate links from link rel=alternate tags. items: $ref: '#/components/schemas/PageMetadataAlternate' jsonLd: type: array description: JSON-LD structured data blocks parsed from the page. items: type: object additionalProperties: true additionalMeta: type: object description: >- Additional non-social meta tags not promoted to top-level metadata fields. additionalProperties: $ref: '#/components/schemas/PageMetadataValue' required: - sourceUrl - finalUrl KeyMetadata: type: object description: >- Metadata about the API key used for the request. Included in every response whenever a valid API key is provided, even when the response status is not 200. properties: credits_consumed: type: integer description: The number of credits consumed by this request. credits_remaining: type: integer description: >- The number of credits remaining for your organization after this request. required: - credits_consumed - credits_remaining PageMetadataValue: oneOf: - type: string - type: array items: type: string PageMetadataAlternate: type: object properties: href: type: string description: Resolved alternate URL. hreflang: type: string description: Language or locale for the alternate URL, when present. type: type: string description: Alternate resource MIME type, when present. title: type: string description: Alternate resource title, when present. required: - href additionalProperties: false responses: RateLimited: description: Rate limit exceeded headers: Retry-After: description: Seconds until the per-minute rate limit window resets schema: type: integer minimum: 1 maximum: 60 content: application/json: schema: type: object properties: message: type: string description: Error message error_code: type: string enum: - RATE_LIMITED description: Error code indicating the rate limit was exceeded key_metadata: $ref: '#/components/schemas/KeyMetadata' required: - message - error_code securitySchemes: bearerAuth: type: http scheme: bearer ````