Skip to main content
Context.dev supports gzip response compression across the API. Send Accept-Encoding: gzip with your request, or use a client that sends it automatically, and Context.dev returns compressed JSON when the response is large enough to benefit. This is most useful for response-heavy endpoints like /web/scrape/html, /web/scrape/markdown, and /web/crawl, all covered in the Scrape Websites guide. In production benchmarks on scrape responses, gzip reduced bytes on the wire by about 82%. Large payloads, especially 500KB and above, saw a median latency improvement of about 70ms, while smaller responses were roughly neutral. For brand, quality, extraction, and classification-style endpoints, gzip is still supported as an option. The benefit depends on how large the response is: small JSON payloads may not move much, while larger AI or crawl-derived responses can benefit from the same transfer-size reduction.

​
Enable gzip

Most HTTP clients already request gzip and decompress the response automatically. The main rule is: don’t override Accept-Encoding with identity unless you are deliberately measuring an uncompressed baseline. 
curl --compressed -G https://api.context.dev/v1/web/scrape/html \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "url=https://example.com"
With requests and the official SDKs, the decoded JSON is what your application sees. The compression and decompression happen at the HTTP layer.

​
Verify compression

To confirm gzip is being negotiated, inspect the response headers: 
curl -s -o /dev/null -D - -G https://api.context.dev/v1/web/scrape/markdown \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Accept-Encoding: gzip" \
  --data-urlencode "url=https://example.com"
Look for: 
content-encoding: gzip
If you see no content-encoding header, the response may be small enough that compression is not worth applying, or your client or proxy may have already decompressed it before exposing headers.

​
Where compression helps most

Gzip improves transfer size for any compressible JSON response, but the latency impact depends on payload size and network conditions.
Response shapeExpected impact
Large rendered HTMLHighest benefit. HTML compresses well, and /web/scrape/html can return hundreds of KB or multiple MB.
Full-site crawl responsesHigh benefit. /web/crawl can return many Markdown documents in one JSON response.
Markdown scrape responsesModerate benefit. Markdown is usually smaller than HTML, but large pages still benefit.
Brand and quality-style JSON responsesOptional. Use gzip when available, but expect smaller gains unless the response is large.
Very small responsesUsually neutral. The response is already small, so transfer savings are limited.
The benchmarked scrape responses had a median compression ratio of 6.5x for HTML and 4.2x for Markdown. On slower, mobile, metered, or distant networks, the same byte reduction can translate into a larger user-visible latency win than it does from a fast server-to-server connection.

​
Keep connections warm

Compression reduces bytes on the wire, but connection setup can still dominate latency. Reuse HTTP connections instead of creating a fresh client for every request: 
import axios from "axios";
import https from "node:https";

const api = axios.create({
  baseURL: "https://api.context.dev/v1",
  headers: {
    Authorization: `Bearer ${process.env.CONTEXT_DEV_API_KEY}`,
    "Accept-Encoding": "gzip",
  },
  httpsAgent: new https.Agent({ keepAlive: true }),
});

const { data } = await api.get("/web/scrape/html", {
  params: { url: "https://example.com" },
});
For bulk scrape or crawl jobs, a reused client plus gzip is usually the best default: fewer transferred bytes, less network time on large responses, and less work for your application to move data around.

​
Combine with payload controls

Compression works best alongside endpoint parameters that avoid returning bytes you do not need:
  • Use useMainContentOnly=true when you only need the main article or page body.
  • Prefer /web/scrape/markdown over /web/scrape/html when plain text is enough.
  • Keep includeImages=false and shortenBase64Images=true for Markdown scrapes unless image references are required.
  • Use includeSelectors and excludeSelectors to narrow scrape output to the relevant page regions.
  • Use maxAgeMs for repeated scrapes so cached responses can be served quickly.

Scrape websites

Use scrape and crawl endpoints that benefit most from response compression.

Integration best practices

Production patterns for caching, timeouts, retries, and background jobs.

Prefetching

Warm brand lookups before user-facing requests.

Troubleshooting

Diagnose timeouts, retries, and slow requests.