Page content

Crawl options live at the top level of the request. Their names differ from Scrape: use useMainContentOnly and waitForMs, not Scrape’s sharedParams fields. Unknown Crawl fields can be ignored, so check names carefully.

Keep article content

Send this body to POST /web/crawl. See the Quickstart for authentication and SDK setup.

{
  "url": "https://example.com/blog",
  "maxPages": 10,
  "useMainContentOnly": true,
  "excludeSelectors": [
    "nav",
    "footer"
  ],
  "includeLinks": true,
  "includeImages": false
}

Field	Default	Effect
`useMainContentOnly`	`false`	Remove detected page chrome.
`includeSelectors`, `excludeSelectors`	None	Keep or remove matching elements; exclusions win.
`includeLinks`	`true`	Keep link destinations.
`includeImages`	`false`	Include image references.
`shortenBase64Images`	`true`	Replace inline base64 payloads with placeholders.
`includeFrames`	`false`	Read iframe content.
`waitForMs`	3,500	Wait before extraction.
`settleAnimations`	`false`	Wait for CSS animations.
`maxAgeMs`	One day	Accept cached page content; `0` fetches fresh.

Use pdf: {"shouldParse":true,"ocr":true,"start":1,"end":5} for selected PDF pages. Page numbers are inclusive and start at 1. OCR recovers scanned pages while preserving existing text. country selects the fetch location, zdr applies zero data retention, and tags label the request in usage logs.

Get started

Web data

Company and people data

Automation

Account and billing

Production

Keep article content

​Keep article content

Keep article content