> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Map URLs

> Map all URLs a website has using Context.dev's index, with available page titles, descriptions, keywords, and languages.

<Badge color="blue">1 Credit</Badge> <Badge color="orange">With search: 2 Credits</Badge>

Map a website's URLs using Context.dev's index. Filters and `maxLinks` bound the returned inventory. URLs without stored page metadata return immediately with only `url` and are queued for background enrichment. Requests with zero data retention or target credentials return URLs only and skip enrichment. See [Map website URLs](/guides/discover-website-urls) for filters, limits, and choosing what to scrape next.


## OpenAPI

````yaml GET /web/urls
openapi: 3.1.0
info:
  title: Context.dev API
  description: API for retrieving context data from any website
  version: 1.0.0
servers:
  - url: https://api.context.dev/v1
security: []
tags:
  - name: Webhooks
    description: Inspect and retry webhook deliveries. These endpoints cost no credits.
  - name: Logs
    description: >-
      Read your organization's API request logs to debug failed calls. These
      endpoints cost no credits and use a separate rate limit.
  - name: Batch
    description: Scrape many pages or crawl a site asynchronously.
  - name: Monitors
    description: >-
      Monitor pages, sitemaps, and extracted website data for exact or semantic
      changes. Webhook payloads are documented by the
      MonitorsChangeDetectedWebhookPayload and
      MonitorsRunCompletedWebhookPayload schemas.
  - name: News
    description: >-
      Search live first-party RSS and free historical news data by company
      identity.
  - name: Answers
    description: Answer a research task from the live web in the JSON shape you ask for.
paths:
  /web/urls:
    get:
      tags:
        - Web Scraping
      summary: Map website URLs
      description: >-
        Maps all URLs a website has using Context.dev's index. Each URL includes
        its available title, description, keywords, and language. URLs without
        stored enrichment are returned immediately with only the URL and queued
        for background HTML scraping, so later requests can include their
        metadata. Responses are never cached as a whole; every request reads the
        current per-URL enrichment. Zero data retention and credential-bearing
        requests return URLs without reading or storing shared enrichment or
        queuing background scrapes. Costs 1 credit, or 2 credits with search.
      parameters:
        - schema:
            type: string
            minLength: 3
            description: Domain to map using Context.dev's index.
          required: true
          description: Domain to map using Context.dev's index.
          name: domain
          in: query
        - schema:
            type: boolean
            default: false
            description: >-
              When true, include indexed URLs on subdomains of the requested
              domain. Defaults to false.
          required: false
          description: >-
            When true, include indexed URLs on subdomains of the requested
            domain. Defaults to false.
          name: includeSubdomains
          in: query
        - schema:
            type: integer
            minimum: 1
            maximum: 100000
            default: 10000
            description: >-
              Maximum number of URLs to return from the index. Defaults to
              10,000. Minimum is 1, maximum is 100,000.
          required: false
          description: >-
            Maximum number of URLs to return from the index. Defaults to 10,000.
            Minimum is 1, maximum is 100,000.
          name: maxLinks
          in: query
        - schema:
            type: string
            maxLength: 256
            description: >-
              Optional RE2-compatible regex pattern. Only URLs matching this
              pattern are returned and counted against maxLinks.
            example: ^https?://[^/]+/blog/
          required: false
          description: >-
            Optional RE2-compatible regex pattern. Only URLs matching this
            pattern are returned and counted against maxLinks.
          name: urlRegex
          in: query
        - schema:
            type: string
            minLength: 2
            maxLength: 200
            description: >-
              Optional search phrase. Filters indexed URLs to pages about that
              phrase, most relevant first. The request costs 2 credits instead
              of 1.
            example: help center and troubleshooting articles
          required: false
          description: >-
            Optional search phrase. Filters indexed URLs to pages about that
            phrase, most relevant first. The request costs 2 credits instead of
            1.
          name: search
          in: query
        - schema:
            type: object
            additionalProperties:
              type: string
              maxLength: 8192
              pattern: ^[^\r\n]*$
            description: >-
              Optional outbound HTTP headers forwarded only to the target URL,
              sent as deep-object query params such as headers[X-Custom]=value.
              When provided, caching is bypassed: the result is neither read
              from nor written to cache.
          required: false
          description: >-
            Optional outbound HTTP headers forwarded only to the target URL,
            sent as deep-object query params such as headers[X-Custom]=value.
            When provided, caching is bypassed: the result is neither read from
            nor written to cache.
          name: headers
          in: query
        - schema:
            $ref: '#/components/schemas/LenientPartialTimeout'
          required: false
          description: >-
            Optional request deadline and behavior on timeout. For GET requests,
            use timeoutOpts[milliseconds]=30000&timeoutOpts[behavior]=fail or a
            JSON-encoded timeoutOpts object.
          style: deepObject
          explode: true
          name: timeoutOpts
          in: query
        - schema:
            type: string
            enum:
              - enabled
              - disabled
            default: disabled
            description: >-
              Set to enabled to bypass shared caches and omit request and
              response content from retained usage logs. Asset uploads are
              skipped, so hosted image URLs are omitted. Requires zero data
              retention to be enabled for your organization (contact
              support@context.dev), otherwise the request fails with
              ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR:
              true.
          required: false
          description: >-
            Set to enabled to bypass shared caches and omit request and response
            content from retained usage logs. Asset uploads are skipped, so
            hosted image URLs are omitted. Requires zero data retention to be
            enabled for your organization (contact support@context.dev),
            otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR
            responses include X-Context-ZDR: true.
          name: zdr
          in: query
        - $ref: '#/components/parameters/RequestTags'
      responses:
        '200':
          description: Mapped URLs with available page metadata
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
            X-Context-ZDR:
              description: >-
                Present with the value true when zero data retention was
                requested and honored.
              schema:
                type: string
                enum:
                  - 'true'
            Cache-Control:
              schema:
                type: string
                enum:
                  - no-store
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                    enum:
                      - true
                  domain:
                    type: string
                  urls:
                    type: array
                    items:
                      type: object
                      properties:
                        url:
                          type: string
                        title:
                          type: string
                        description:
                          type: string
                        keywords:
                          type: array
                          items:
                            type: string
                        language:
                          type: string
                      required:
                        - url
                  partial:
                    type: boolean
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - success
                  - domain
                  - urls
                  - request_id
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          description: Unauthorized - Invalid or missing API key
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Error message
                  error_code:
                    type: string
                    enum:
                      - UNAUTHORIZED
                    description: Error code indicating unauthorized access
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - request_id
        '403':
          description: Forbidden - Insufficient permissions or usage limit exceeded
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Error message
                  error_code:
                    type: string
                    enum:
                      - FORBIDDEN
                      - USAGE_EXCEEDED
                      - DISABLED
                      - INSUFFICIENT_PERMISSIONS
                    description: Error code indicating forbidden access
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - request_id
        '408':
          description: Request timeout
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Timeout error message
                  error_code:
                    type: string
                    enum:
                      - REQUEST_TIMEOUT
                    description: Error code indicating request timeout
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - request_id
        '429':
          $ref: '#/components/responses/RateLimited'
        '500':
          description: Internal server error
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Error message
                  error_code:
                    type: string
                    enum:
                      - INTERNAL_ERROR
                    description: Error code indicating internal server error
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - request_id
      security:
        - bearerAuth: []
components:
  schemas:
    LenientPartialTimeout:
      type: object
      properties:
        milliseconds:
          type: integer
          minimum: 1
          maximum: 300000
          description: 'Request deadline in milliseconds. Maximum: 300000 (5 minutes).'
        behavior:
          type: string
          enum:
            - fail
            - return-partial
          default: fail
          description: >-
            What to do at the deadline. "fail" returns 408 REQUEST_TIMEOUT
            without charging credits. "return-partial" returns usable results
            collected so far; if none are available, the request still fails
            without charging credits. Partial results are not cached as complete
            results.
      required:
        - milliseconds
      additionalProperties: false
      description: >-
        Optional request deadline and behavior on timeout. For GET requests, use
        timeoutOpts[milliseconds]=30000&timeoutOpts[behavior]=fail or a
        JSON-encoded timeoutOpts object.
    KeyMetadata:
      type: object
      properties:
        credits_consumed:
          type: integer
          description: Credits used by this request.
        credits_remaining:
          type: integer
          description: Credits remaining for your organization.
      required:
        - credits_consumed
        - credits_remaining
      description: Credit usage, included whenever a valid API key is provided.
    RequestId:
      type: string
      format: uuid
      description: >-
        Unique id of this API call, also sent in the X-Request-Id response
        header. Quote it when contacting support about a failed request.
      example: 3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91
    RequestTags:
      type: array
      items:
        type: string
        minLength: 1
        maxLength: 50
      maxItems: 20
      description: >-
        Optional tags for tracking usage. Up to 20 tags, each 1 to 50
        characters.
      example:
        - production
        - team-alpha
    ErrorResponse:
      type: object
      properties:
        message:
          type: string
          description: Error message
        error_code:
          type: string
          enum:
            - INTERNAL_ERROR
            - VALID
            - NOT_FOUND
            - FORBIDDEN
            - USAGE_EXCEEDED
            - RATE_LIMITED
            - UNAUTHORIZED
            - DISABLED
            - PAID_PLAN_REQUIRED
            - INSUFFICIENT_PERMISSIONS
            - TIMEOUT_EXCEEDS_MAXIMUM
            - TIMEOUT_TOO_SHORT_FOR_WAIT
            - WEBSITE_ACCESS_ERROR
            - WEBSITE_BLOCKED
            - WEBSITE_NOT_FOUND
            - PDF_SKIPPED
            - PDF_IMAGES_ONLY
            - EXTERNAL_PROVIDER_ERROR
            - INPUT_VALIDATION_ERROR
            - ZDR_NOT_SUPPORTED
            - ZDR_NOT_ENABLED
            - FREE_EMAIL_DETECTED
            - DISPOSABLE_EMAIL_DETECTED
            - REQUEST_TIMEOUT
            - COLD_DOMAIN_TIMEOUT_TOO_LOW
            - UNSUPPORTED_CONTENT
            - CONTENT_TOO_LARGE
            - MONITOR_PAUSED
            - MONITOR_NO_WEBHOOK
            - COLLECTION_PAUSED
            - MONITOR_LIMIT_EXCEEDED
            - SEARCH_UNAVAILABLE
            - BATCH_LIMIT_EXCEEDED
            - BATCH_NOT_CANCELLABLE
            - BATCH_NOT_COMPLETED
            - IDEMPOTENCY_KEY_CONFLICT
            - DELIVERY_IN_PROGRESS
            - DELIVERY_ALREADY_DELIVERED
            - DELIVERY_EXPIRED
            - DELIVERY_CANCELLED
          description: Error code indicating the type of error
        required_permission:
          type: string
          enum:
            - logs:read
            - data:execute
            - monitors:read
            - monitors:write
            - batches:read
            - batches:write
            - webdbs:read
            - webdbs:write
          description: >-
            Permission required for this request when error_code is
            INSUFFICIENT_PERMISSIONS. Manage (write) also grants read access to
            the same resource group.
        key_metadata:
          $ref: '#/components/schemas/KeyMetadata'
        request_id:
          $ref: '#/components/schemas/RequestId'
      required:
        - request_id
  parameters:
    RequestTags:
      name: tags
      in: query
      required: false
      style: form
      explode: false
      schema:
        $ref: '#/components/schemas/RequestTags'
      description: >-
        Comma-separated tags for tracking request usage. Up to 20 tags, each
        1-50 characters.
      example: production,team-alpha
  headers:
    RequestId:
      description: >-
        Unique id of this API call, sent on every response and mirrored as
        request_id in JSON bodies. Quote it when contacting support.
      schema:
        type: string
        format: uuid
    RateLimitLimit:
      description: Maximum requests per minute.
      schema:
        type: integer
        minimum: 1
    RateLimitRemaining:
      description: Requests remaining in the current minute.
      schema:
        type: integer
        minimum: 0
    RateLimitReset:
      description: Unix timestamp in seconds when the rate limit resets.
      schema:
        type: integer
  responses:
    BadRequest:
      description: Bad request
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
      headers:
        X-Request-Id:
          $ref: '#/components/headers/RequestId'
        X-RateLimit-Limit:
          $ref: '#/components/headers/RateLimitLimit'
        X-RateLimit-Remaining:
          $ref: '#/components/headers/RateLimitRemaining'
        X-RateLimit-Reset:
          $ref: '#/components/headers/RateLimitReset'
    RateLimited:
      description: Rate limit exceeded
      headers:
        X-Request-Id:
          $ref: '#/components/headers/RequestId'
        X-RateLimit-Limit:
          $ref: '#/components/headers/RateLimitLimit'
        X-RateLimit-Remaining:
          $ref: '#/components/headers/RateLimitRemaining'
        X-RateLimit-Reset:
          $ref: '#/components/headers/RateLimitReset'
        Retry-After:
          description: Seconds until the per-minute rate limit window resets
          schema:
            type: integer
            minimum: 1
            maximum: 60
      content:
        application/json:
          schema:
            type: object
            properties:
              message:
                type: string
                description: Error message
              error_code:
                type: string
                enum:
                  - RATE_LIMITED
                description: Error code indicating the rate limit was exceeded
              key_metadata:
                $ref: '#/components/schemas/KeyMetadata'
              request_id:
                $ref: '#/components/schemas/RequestId'
            required:
              - request_id
              - message
              - error_code
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: >-
        Bearer authentication header of the form `Bearer <API_KEY>`. Keys have
        full access by default.

````