Scrape a webpage

Use the Web API when you have one URL and need its readable content. The Markdown endpoint returns content ready for search, RAG, or an LLM; the HTML endpoint returns the rendered document.

Need more than one page? Crawl a website. Need JSON fields rather than page content? Extract structured data.

Prerequisites

Create an API key in the dashboard and keep it in a server-side environment variable:

export CONTEXT_DEV_API_KEY="ctxt_secret_..."

Install an SDK from the Quickstart, or use the HTTPS example below without an SDK.

Get Markdown

Pass a complete http:// or https:// URL. useMainContentOnly removes detected navigation, sidebars, and footers so the result focuses on the page body.

curl --get https://api.context.dev/v1/web/scrape/markdown \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "useMainContentOnly=true"

import ContextDev from "context.dev";

const client = new ContextDev({
  apiKey: process.env.CONTEXT_DEV_API_KEY,
});

const page = await client.web.webScrapeMd({
  url: "https://example.com",
  useMainContentOnly: true,
});

console.log(page.markdown);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

page = client.web.web_scrape_md(
    url="https://example.com",
    use_main_content_only=True,
)

print(page.markdown)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

page = client.web.web_scrape_md(
  url: "https://example.com",
  use_main_content_only: true,
)

puts page.markdown

package main

import (
    "context"
    "fmt"
    "os"

    contextdev "github.com/context-dot-dev/context-go-sdk/v2"
    "github.com/context-dot-dev/context-go-sdk/v2/option"
    "github.com/context-dot-dev/context-go-sdk/v2/packages/param"
)

func main() {
    client := contextdev.NewClient(
        option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
    )

    page, err := client.Web.WebScrapeMd(context.Background(), contextdev.WebWebScrapeMdParams{
        URL: "https://example.com",
        UseMainContentOnly: param.NewOpt(true),
    })
    if err != nil {
        panic(err)
    }

    fmt.Println(page.Markdown)
}

<?php

require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$page = $client->web->webScrapeMd(
    url: 'https://example.com',
    useMainContentOnly: true,
);

echo $page->markdown, PHP_EOL;

A successful scrape costs 1 credit, or 2 credits with browser actions. PDF OCR adds 1 credit per page recovered. A successful response includes the requested URL, Markdown, page metadata, and cache metadata:

{
  "success": true,
  "url": "https://example.com",
  "markdown": "# Example Domain\n\nThis domain is for use in examples.",
  "contentLength": 55,
  "metadata": {
    "sourceUrl": "https://example.com",
    "finalUrl": "https://example.com",
    "title": "Example Domain"
  },
  "cache_metadata": {
    "status": "miss",
    "age_ms": 0
  }
}

Treat the API reference as the authority for the complete response schema.

Get rendered HTML

Use GET /web/scrape/html when downstream code needs DOM markup rather than Markdown.

curl --get https://api.context.dev/v1/web/scrape/html \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "url=https://example.com"

const page = await client.web.webScrapeHTML({
  url: "https://example.com",
});

console.log(page.html);

page = client.web.web_scrape_html(url="https://example.com")
print(page.html)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

page = client.web.web_scrape_html(url: "https://example.com")
puts page.html

package main

import (
    "context"
    "fmt"
    "os"

    contextdev "github.com/context-dot-dev/context-go-sdk/v2"
    "github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
    client := contextdev.NewClient(
        option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
    )

    page, err := client.Web.WebScrapeHTML(context.Background(), contextdev.WebWebScrapeHTMLParams{
        URL: "https://example.com",
    })
    if err != nil {
        panic(err)
    }

    fmt.Println(page.HTML)
}

<?php

require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$page = $client->web->webScrapeHTML(url: 'https://example.com');
echo $page->html, PHP_EOL;

You can also set includeHTML=true on a Markdown request when you need both representations from the same scrape.

Content controls

Goal	Option	Behavior
Keep the primary article or product content	`useMainContentOnly=true`	Removes detected page chrome.
Preserve images in Markdown	`includeImages=true`	Includes image references in the Markdown result.
Remove a known page region	`excludeSelectors=["nav", ".cookie-banner"]`	Excludes matching DOM subtrees.
Keep only a known page region	`includeSelectors=["article"]`	Converts only matching DOM subtrees.
Include iframe content	`includeFrames=true`	Renders iframe content into the result.

If an include selector matches nothing, the request can still succeed with an empty result. Check contentLength before indexing or summarizing the page.

Handle dynamic or protected pages

Use the least expensive control that produces the page state you need:

Set waitForMs when content appears shortly after load.
Set settleAnimations=true when transitions make captures inconsistent.
Use browser actions when the page requires a click, scroll, or ordered interaction.
Set country when the page varies by country or is unavailable from the default route.

Context.dev can retry blocked requests through different proxy routes. This improves access, but it does not guarantee access to every site. A challenge page, CAPTCHA, or login shell can return WEBSITE_BLOCKED; unsuccessful requests are not billed.

Control freshness and retention

The default maxAgeMs is 86400000 (one day). A matching cached result younger than that value can be returned.

Requirement	Setting
Reuse results for up to one day	Omit `maxAgeMs`.
Fetch the page again	Set `maxAgeMs=0`.
Accept a result up to 30 days old	Set `maxAgeMs=2592000000`.
Bypass shared caches and retained content logs	Set `zdr=enabled`; your organization must have Zero Data Retention enabled.

Forwarding custom headers also bypasses cache reads and writes. Keep secrets out of forwarded headers unless the target site requires them.

Plan for failures

Status	Meaning	Next action
`400`	Invalid input, inaccessible page, blocked page, or skipped PDF	Inspect `error_code`; change the request before retrying validation errors.
`404`	Target page was not found	Confirm the final URL or stop processing that record.
`408`	The request exceeded `timeoutOpts`	Retry outside a user-facing path or allow a larger timeout.
`413`	Target content exceeds 20 MB	Use a smaller source or a purpose-built document workflow.
`415`	Unsupported content type	Use document parsing for supported uploaded files.
`429`	Per-minute limit exceeded	Honor `Retry-After` and retry with backoff.
`500`	Internal failure	Retry with bounded backoff.

See Troubleshooting for stable error codes and retry guidance.

Next steps

Build live web tools for an agent to turn selected pages into bounded, citable evidence with explicit read failures.

Discover website URLs

Find candidate URLs in public sitemaps.

Crawl a website

Follow links and collect content from multiple pages.

Extract page images

Get image sources, dimensions, and optional classification.

Extract structured data

Return fields shaped by your JSON Schema.

Scrape websites in batches

Process large or asynchronous scraping workloads.

Production best practices

Prepare retries, caching, and error handling for user-facing requests.

With timeoutOpts.behavior: "return-partial", a scrape can return finalDOMState: "still-loading" when the deadline arrives before the requested waits finish. Treat that content as incomplete. See Timeouts and partial results for minimum budgets, billing, and migration guidance.

Get started

Web data

Brand data

Automation

Account and billing

Production

Prerequisites

Get Markdown

Get rendered HTML

Content controls

Handle dynamic or protected pages

Control freshness and retention

Plan for failures

Next steps

Discover website URLs

Crawl a website

Extract page images

Extract structured data

Scrape websites in batches

Production best practices

​Prerequisites

​Get Markdown

​Get rendered HTML

​Content controls

​Handle dynamic or protected pages

​Control freshness and retention

​Plan for failures

​Next steps

Discover website URLs

Crawl a website

Extract page images

Extract structured data

Scrape websites in batches

Production best practices

Prerequisites

Get Markdown

Get rendered HTML

Content controls

Handle dynamic or protected pages

Control freshness and retention

Plan for failures

Next steps