Scope and limits

Crawl starts at url and follows eligible links. Send these fields to POST /web/crawl.

Limit a crawl

Send this body to POST /web/crawl. See the Quickstart for authentication and SDK setup.

{
  "url": "https://example.com/docs",
  "maxPages": 50,
  "maxDepth": 2,
  "urlRegex": "^https://example[.]com/docs/",
  "stopAfterMs": 60000
}

Field	Default	Limit
`maxPages`	100	1–500 pages.
`maxDepth`	Unset	Limit link depth from the start URL.
`urlRegex`	None	RE2-compatible URL filter, up to 256 characters.
`followSubdomains`	`false`	Permit child hosts.
`stopAfterMs`	80,000	Soft budget, 10,000–110,000 ms.
`timeoutOpts`	Unset	Request deadline; minimum 1,000 ms.

The soft budget is checked after each page finishes, so it can be exceeded by the current page. timeoutOpts sets the overall deadline; choose return-partial when completed pages are useful. A page cap is a maximum, not a promise: a site may expose fewer eligible links. For more than 500 pages, use Large crawls.

Get started

Web data

Company and people data

Automation

Account and billing

Production

Limit a crawl

​Limit a crawl

Limit a crawl