Compression - Context.dev

Context.dev supports gzip response compression across the API. Send Accept-Encoding: gzip with your request, or use a client that sends compression headers automatically, and Context.dev returns compressed JSON when the response is large enough to benefit. This is most useful for response-heavy endpoints like /web/scrape/html, /web/scrape/markdown, and /web/crawl, all covered in the Scrape Websites guide. In production benchmarks on scrape responses, gzip reduced bytes on the wire by about 82%. Large payloads, especially 500KB and above, saw a median latency improvement of about 70ms, while smaller responses were roughly neutral. For brand, quality, extraction, and classification-style endpoints, gzip is still supported as an option. The benefit depends on how large the response is: small JSON payloads may not move much, while larger AI or crawl-derived responses can benefit from the same transfer-size reduction.

Enable gzip

Most HTTP clients already request compressed responses and decompress them automatically. The main rules are:

Don’t override Accept-Encoding with identity unless you are deliberately measuring an uncompressed baseline.
Let Ruby Net::HTTP and Go net/http manage Accept-Encoding; if you set that header manually in those clients, they return compressed bytes and you must decompress them yourself.
Treat the decoded JSON as the application contract. Some clients expose content-encoding: gzip, some negotiate br, and some strip the header after decompression.

All examples below read the API key from CONTEXT_DEV_API_KEY and request /web/scrape/markdown with automatic decompression. No manual gunzip step is needed. Run these from a backend process; never expose the secret API key in browser code.

curl --compressed -G https://api.context.dev/v1/web/scrape/markdown \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "useMainContentOnly=true"

type ScrapeMarkdownResponse = {
  success: boolean;
  markdown: string;
};

const apiKey = process.env.CONTEXT_DEV_API_KEY;

if (!apiKey) {
  throw new Error("CONTEXT_DEV_API_KEY is required");
}

const response = await fetch(
  "https://api.context.dev/v1/web/scrape/markdown?url=https%3A%2F%2Fexample.com&useMainContentOnly=true",
  {
    headers: {
      Authorization: `Bearer ${apiKey}`,
    },
  },
);

if (!response.ok) {
  throw new Error(`Context.dev returned ${response.status}: ${await response.text()}`);
}

const data = (await response.json()) as ScrapeMarkdownResponse;
console.log(data.markdown);

import axios from "axios";

const apiKey = process.env.CONTEXT_DEV_API_KEY;

if (!apiKey) {
  throw new Error("CONTEXT_DEV_API_KEY is required");
}

const { data } = await axios.get(
  "https://api.context.dev/v1/web/scrape/markdown",
  {
    params: {
      url: "https://example.com",
      useMainContentOnly: true,
    },
    headers: {
      Authorization: `Bearer ${apiKey}`,
    },
  },
);

console.log(data.markdown);

import os
import requests

api_key = os.environ["CONTEXT_DEV_API_KEY"]

response = requests.get(
    "https://api.context.dev/v1/web/scrape/markdown",
    params={"url": "https://example.com", "useMainContentOnly": "true"},
    headers={"Authorization": f"Bearer {api_key}"},
    timeout=60,
)
response.raise_for_status()

data = response.json()
print(data["markdown"])

require "json"
require "net/http"
require "uri"

api_key = ENV.fetch("CONTEXT_DEV_API_KEY")

uri = URI("https://api.context.dev/v1/web/scrape/markdown")
uri.query = URI.encode_www_form(
  url: "https://example.com",
  useMainContentOnly: "true",
)

request = Net::HTTP::Get.new(uri)
request["Authorization"] = "Bearer #{api_key}"

response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
  http.request(request)
end

unless response.is_a?(Net::HTTPSuccess)
  raise "Context.dev returned #{response.code}: #{response.body}"
end

data = JSON.parse(response.body)
puts data.fetch("markdown")

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "net/url"
    "os"
    "time"
)

type ScrapeMarkdownResponse struct {
    Success  bool   `json:"success"`
    Markdown string `json:"markdown"`
}

func main() {
    apiKey := os.Getenv("CONTEXT_DEV_API_KEY")
    if apiKey == "" {
        panic("CONTEXT_DEV_API_KEY is required")
    }

    params := url.Values{}
    params.Set("url", "https://example.com")
    params.Set("useMainContentOnly", "true")

    req, err := http.NewRequest(
        http.MethodGet,
        "https://api.context.dev/v1/web/scrape/markdown?"+params.Encode(),
        nil,
    )
    if err != nil {
        panic(err)
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)

    client := &http.Client{Timeout: 60 * time.Second}
    response, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer response.Body.Close()

    if response.StatusCode < 200 || response.StatusCode >= 300 {
        panic(fmt.Sprintf("Context.dev returned %d", response.StatusCode))
    }

    var data ScrapeMarkdownResponse
    if err := json.NewDecoder(response.Body).Decode(&data); err != nil {
        panic(err)
    }

    fmt.Println(data.Markdown)
}

For Node Axios and Python requests, install the client library first:

npm install axios

pip install requests

With these clients, the decoded JSON is what your application sees. Compression and decompression happen at the HTTP layer.

Verify compression

To confirm gzip is being negotiated, inspect the response headers with a client that does not hide the wire-level response:

curl -s -o /dev/null -D - -G https://api.context.dev/v1/web/scrape/markdown \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Accept-Encoding: gzip" \
  --data-urlencode "url=https://example.com"

Look for:

content-encoding: gzip

If you see no content-encoding header in application code, the response may be small enough that compression is not worth applying, or your client may have already decompressed it before exposing headers. That is normal for clients like Axios, Ruby Net::HTTP, and Go net/http. Node’s built-in fetch may expose gzip or br depending on what it negotiated; the response body is still decoded before response.json() runs.

Where compression helps most

Gzip improves transfer size for any compressible JSON response, but the latency impact depends on payload size and network conditions.

Response shape	Expected impact
Large rendered HTML	Highest benefit. HTML compresses well, and `/web/scrape/html` can return hundreds of KB or multiple MB.
Full-site crawl responses	High benefit. `/web/crawl` can return many Markdown documents in one JSON response.
Markdown scrape responses	Moderate benefit. Markdown is usually smaller than HTML, but large pages still benefit.
Brand and quality-style JSON responses	Optional. Use gzip when available, but expect smaller gains unless the response is large.
Very small responses	Usually neutral. The response is already small, so transfer savings are limited.

The benchmarked scrape responses had a median compression ratio of 6.5x for HTML and 4.2x for Markdown. On slower, mobile, metered, or distant networks, the same byte reduction can translate into a larger user-visible latency win than it does from a fast server-to-server connection.

Keep connections warm

Compression reduces bytes on the wire, but connection setup can still dominate latency. Reuse HTTP connections instead of creating a fresh client for every request:

import axios from "axios";
import https from "node:https";

const api = axios.create({
  baseURL: "https://api.context.dev/v1",
  headers: {
    Authorization: `Bearer ${process.env.CONTEXT_DEV_API_KEY}`,
    "Accept-Encoding": "gzip",
  },
  httpsAgent: new https.Agent({ keepAlive: true }),
});

const { data } = await api.get("/web/scrape/html", {
  params: { url: "https://example.com" },
});

import os
import requests

session = requests.Session()
session.headers.update({
    "Authorization": f"Bearer {os.environ['CONTEXT_DEV_API_KEY']}",
    "Accept-Encoding": "gzip",
})

response = session.get(
    "https://api.context.dev/v1/web/scrape/html",
    params={"url": "https://example.com"},
)

For bulk scrape or crawl jobs, a reused client plus gzip is usually the best default: fewer transferred bytes, less network time on large responses, and less work for your application to move data around.

Combine with payload controls

Compression works best alongside endpoint parameters that avoid returning bytes you do not need:

Use useMainContentOnly=true when you only need the main article or page body.
Prefer /web/scrape/markdown over /web/scrape/html when plain text is enough.
Keep includeImages=false and shortenBase64Images=true for Markdown scrapes unless image references are required.
Use includeSelectors and excludeSelectors to narrow scrape output to the relevant page regions.
Use maxAgeMs for repeated scrapes so cached responses can be served quickly.

Scrape websites

Use scrape and crawl endpoints that benefit most from response compression.

Integration best practices

Production patterns for caching, timeouts, retries, and background jobs.

Prefetching

Warm brand lookups before user-facing requests.

Troubleshooting

Diagnose timeouts, retries, and slow requests.

​Enable gzip

​Verify compression

​Where compression helps most

​Keep connections warm

​Combine with payload controls

​Related resources