Skip to main content
GET
Map website URLs
1 Credit With search: 2 Credits Map a website’s URLs using Context.dev’s index. Filters and maxLinks bound the returned inventory. URLs without stored page metadata return immediately with only url and are queued for background enrichment. Requests with zero data retention or target credentials return URLs only and skip enrichment. See Map website URLs for filters, limits, and choosing what to scrape next.

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <API_KEY>. Keys have full access by default.

Query Parameters

domain
string
required

Domain to map using Context.dev's index.

Minimum string length: 3
includeSubdomains
boolean
default:false

When true, include indexed URLs on subdomains of the requested domain. Defaults to false.

Maximum number of URLs to return from the index. Defaults to 10,000. Minimum is 1, maximum is 100,000.

Required range: 1 <= x <= 100000
urlRegex
string

Optional RE2-compatible regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.

Maximum string length: 256
Example:

"^https?://[^/]+/blog/"

Optional search phrase. Filters indexed URLs to pages about that phrase, most relevant first. The request costs 2 credits instead of 1.

Required string length: 2 - 200
Example:

"help center and troubleshooting articles"

headers
object

Optional outbound HTTP headers forwarded only to the target URL, sent as deep-object query params such as headers[X-Custom]=value. When provided, caching is bypassed: the result is neither read from nor written to cache.

timeoutOpts
object

Optional request deadline and behavior on timeout. For GET requests, use timeoutOpts[milliseconds]=30000&timeoutOpts[behavior]=fail or a JSON-encoded timeoutOpts object.

zdr
enum<string>
default:disabled

Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Asset uploads are skipped, so hosted image URLs are omitted. Requires zero data retention to be enabled for your organization (contact [email protected]), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.

Available options:
enabled,
disabled
tags
string[]

Comma-separated tags for tracking request usage. Up to 20 tags, each 1-50 characters. Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

Maximum array length: 20
Required string length: 1 - 50
Example:

Response

Mapped URLs with available page metadata

success
enum<boolean>
required
Available options:
true
domain
string
required
urls
object[]
required
request_id
string<uuid>
required

Unique id of this API call, also sent in the X-Request-Id response header. Quote it when contacting support about a failed request.

Example:

"3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91"

partial
boolean
key_metadata
object

Credit usage, included whenever a valid API key is provided.