Access policies

An access policy is a list of domains your requests can or can’t access. Requests to blocked websites are refused, and content from them is never returned or charged.

Access policies are enabled per organization. Contact us to turn them on. Until then, the access policy endpoints and requests that list policies return 403 ACCESS_POLICIES_NOT_ENABLED.

Create a policy

Send this body to POST /v1/access-policies. See the Quickstart for authentication.

{
  "name": "Research sources",
  "allowed_domains": ["arxiv.org", "*.gov"]
}

Use allowed_domains to allow only those domains, or blocked_domains to block them. Send one, not both. Each domain includes its subdomains, and *.gov covers every .gov site. The response includes the policy id, such as pol_3f9c2a7e5b8d4c1f9a6e0b2d7c4f8a1e.

Apply policies to a request

Add policy IDs to policies. This scrape only runs if every listed policy allows the URL:

{
  "url": "https://arxiv.org/abs/2410.01234",
  "policies": ["pol_3f9c2a7e5b8d4c1f9a6e0b2d7c4f8a1e"]
}

On endpoints that take query parameters, send a comma-separated list, such as policies=pol_3f9c2a7e5b8d4c1f9a6e0b2d7c4f8a1e,pol_81ab4d6e2f9c0a7b5e3d1c8f6a4b2e9d. List several policies to combine them. For example, one policy that allows github.com and another that blocks gist.github.com together allow GitHub without gists. Batches apply policies to every page, and monitors apply them to every run.

Apply a policy automatically

Use applies_to to choose where a policy applies without listing it:

`applies_to`	The policy applies to
`request` (default)	Requests that list it in `policies`.
`api_keys`	Every request made with the keys in `api_key_ids`.
`organization`	Every request from your account.

Requests can add more policies but can’t skip API key or organization policies.

Limit an API key

Give an agent or integration a key that can only reach the websites you choose:

{
  "name": "Agent sandbox",
  "allowed_domains": ["arxiv.org", "*.gov"],
  "applies_to": "api_keys",
  "api_key_ids": ["ctxt_keyid_4b7e2f9a1c8d4e6fa3b5c7d9e1f20a4b"]
}

A key’s ID appears as key_id in your request logs.

Apply to your whole account

{
  "name": "No social media",
  "blocked_domains": ["facebook.com", "tiktok.com"],
  "applies_to": "organization"
}

What gets checked

Endpoint	When a website is blocked
Scrape, Crawl, Map, Styleguide, Fonts, Brand, Prefetch	The request returns `403 ACCESS_POLICY_DENIED`. Crawl skips blocked links it finds.
Search, Answers, News	Blocked results are removed. Answers never uses them as sources.
Batches	Blocked URLs are listed in `invalid_urls` and never run. Pages that redirect to a blocked website fail with `ACCESS_POLICY_DENIED`.
Monitors	A blocked target is rejected when you save the monitor. If a policy blocks it later, runs fail with `access_policy_denied`.

Redirects and embedded frames are checked too. Company and people data aren’t affected.

Handle a blocked request

{
  "error_code": "ACCESS_POLICY_DENIED",
  "message": "An access policy blocks gist.github.com.",
  "error_data": {
    "url": "https://gist.github.com/example",
    "denied_by": ["pol_81ab4d6e2f9c0a7b5e3d1c8f6a4b2e9d"]
  }
}

denied_by lists every policy that blocked the URL, including API key and organization policies.

Manage policies

Action	Endpoint
Create	`POST /v1/access-policies`
List	`GET /v1/access-policies`
Retrieve	`GET /v1/access-policies/{policy_id}`
Update	`PATCH /v1/access-policies/{policy_id}`
Delete	`DELETE /v1/access-policies/{policy_id}`

Creating, updating, and deleting policies requires the access_policies:write permission; listing and retrieving them requires access_policies:read. Any key can use them in policies. Don’t grant access_policies:write to a key you limit with a policy, since it could change its own limits. Changes apply right away, including to running batches and monitors. A request that lists a deleted or unknown policy fails with 400.

Allowlists are the strictest option. A blocklist only covers the domains you list, so the same content on another website can still be reached.

Get started

Web data

Entity Enrichment

Automation

Account and billing

Production

Create a policy

Apply policies to a request

Apply a policy automatically

Limit an API key

Apply to your whole account

What gets checked

Handle a blocked request

Manage policies

​Create a policy

​Apply policies to a request

​Apply a policy automatically

​Limit an API key

​Apply to your whole account

​What gets checked

​Handle a blocked request

​Manage policies

Create a policy

Apply policies to a request

Apply a policy automatically

Limit an API key

Apply to your whole account

What gets checked

Handle a blocked request

Manage policies