> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Setting up Context.dev for a user? Follow https://docs.context.dev/agent-quickstart.md. MCP clients sign in with OAuth and need no API key; application code reads CONTEXT_DEV_API_KEY.

# Scrape

> Get page content, images, structured data, and screenshots from one URL.

Scrape visits a URL and returns the outputs you enable in `formats`. Combine formats in one request and check each output’s `success` before using its data.

## Make a request

Set up your API key and SDK in the [Quickstart](/quickstart). This request returns the page’s main content as Markdown.

<CodeGroup>
  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({
    apiKey: process.env.CONTEXT_DEV_API_KEY,
  });

  const page = await client.web.scrape({
    url: "https://example.com",
    formats: { markdown: true },
    sharedParams: { mainContentOnly: true },
  });

  console.log(page.metadata.title);
  console.log(page.markdown.data);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  page = client.web.scrape(
      url="https://example.com",
      formats={"markdown": True},
      shared_params={"main_content_only": True},
  )

  print(page.metadata.title)
  print(page.markdown.data)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  page = client.web.scrape(
    url: "https://example.com",
    formats: {markdown: true},
    shared_params: {main_content_only: true},
  )

  puts page.metadata.title
  puts page.markdown.data
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      contextdev "github.com/context-dot-dev/context-go-sdk/v2"
      "github.com/context-dot-dev/context-go-sdk/v2/option"
  )

  func main() {
      client := contextdev.NewClient(
          option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
      )

      page, err := client.Web.Scrape(context.Background(), contextdev.WebScrapeParams{
          URL:          "https://example.com",
          Formats:      contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
          SharedParams: contextdev.WebScrapeParamsSharedParams{MainContentOnly: contextdev.Bool(true)},
      })
      if err != nil {
          panic(err)
      }

      fmt.Println(page.Metadata.Title)
      fmt.Println(page.Markdown.Data)
  }
  ```

  ```php PHP theme={null}
  <?php

  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $page = $client->web->scrape(
      formats: ['markdown' => true],
      url: 'https://example.com',
      sharedParams: ['mainContentOnly' => true],
  );

  echo $page->metadata->title, PHP_EOL;
  echo $page->markdown->data, PHP_EOL;
  ```

  ```bash cURL theme={null}
  curl https://api.context.dev/v1/web/scrape \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com",
      "formats": { "markdown": true },
      "sharedParams": { "mainContentOnly": true }
    }'
  ```
</CodeGroup>

## Read the response

Every output has `requested`, `success`, and `data`. A successful output has `success: true`. Failed outputs have `success: false` and `data: null`; outputs you did not request have `success: null`. One failed output can coexist with successful ones in an HTTP 200 response.

Abridged response:

```json theme={null}
{
  "url": "https://example.com/",
  "markdown": {"requested": true, "success": true, "data": "# Example Domain"},
  "metadata": {"title": "Example Domain"},
  "cache_metadata": {"status": "miss", "age_ms": 0},
  "request_id": "d7aa1c3a-38d2-4fb5-92c8-0f0cac3efa37"
}
```

`isPartial` flags incomplete results. `metadata` describes the page; `key_metadata` reports request usage. Save `request_id` when investigating failures.

## Formats

| Format       | Output                               | Guide                                |
| ------------ | ------------------------------------ | ------------------------------------ |
| `html`       | Rendered HTML                        | [HTML](/scrape/html)                 |
| `markdown`   | Text with Markdown formatting        | [Markdown](/scrape/markdown)         |
| `screenshot` | Inline image data URL                | [Screenshot](/scrape/screenshot)     |
| `images`     | Image URLs and optional enrichment   | [Images](/scrape/images)             |
| `bytes`      | Original response bytes as base64    | [Bytes](/scrape/bytes)               |
| `parse`      | CSS-selected fields in `parsed.data` | [Parse fields](/scrape/parse-fields) |
| `highlights` | Passages relevant to a question      | [Highlights](/scrape/highlights)     |
| `json`       | An object matching your JSON Schema  | [JSON](/scrape/json)                 |
| `product`    | Product details and availability     | [Product](/scrape/product)           |

## Shape the page visit

Use `sharedParams` for [browser actions](/scrape/browser-actions), [waiting and rendering](/scrape/waiting-and-rendering), [content filtering](/scrape/content-filtering), [location and headers](/scrape/location-and-headers), or [PDF parsing](/scrape/pdfs-and-documents).

Use `maxAgeMs` for [freshness](/scrape/freshness-and-caching), `timeoutOpts` for [deadlines](/scrape/timeouts-and-errors), `zdr` for [zero data retention](/optimization/zero-data-retention), and `tags` for [usage tracking](/optimization/usage-and-logs). The [reference](/api-reference/web-scraping/scrape) lists the complete request and response.

## Choose another API

Use [Map](/map/overview) to find a site’s URLs, [Crawl](/crawl/overview) to read several pages, or [Batches](/batches/overview) to process a URL list asynchronously. [Search](/search/overview) finds pages from a query; [Answers](/answers/overview) researches a question. [Parse](/parse/overview) accepts uploaded files.

## Move from the per-format endpoints

The older routes remain available. New integrations can request several formats through `POST /web/scrape`:

| Earlier route                | Request on Scrape             | Read              |
| ---------------------------- | ----------------------------- | ----------------- |
| `GET /web/scrape/html`       | `formats.html: true`          | `html.data`       |
| `GET /web/scrape/markdown`   | `formats.markdown: true`      | `markdown.data`   |
| `GET /web/scrape/screenshot` | `formats.screenshot: true`    | `screenshot.data` |
| `GET /web/scrape/images`     | `formats.images: true`        | `images.data`     |
| `GET /web/scrape/bytes`      | `formats.bytes: true`         | `bytes.data`      |
| `GET /web/scrape/sitemap`    | Use [Map URLs](/map/overview) | `urls`            |

The option names and response shapes differ. Use the format guides when migrating.