> ## Documentation Index > Fetch the complete documentation index at: https://docs.context.dev/llms.txt > Use this file to discover all available pages before exploring further. # Compression > Use gzip response compression to reduce transfer size and improve latency on large Context.dev responses. Context.dev supports gzip response compression across the API. Send `Accept-Encoding: gzip` with your request, or use a client that sends compression headers automatically, and Context.dev returns compressed JSON when the response is large enough to benefit. This is most useful for response-heavy endpoints like `/web/scrape/html`, `/web/scrape/markdown`, and `/web/crawl`, all covered in the [Scrape Websites guide](/guides/scrape-websites-to-markdown). In production benchmarks on scrape responses, gzip reduced bytes on the wire by about 82%. Large payloads, especially 500KB and above, saw a median latency improvement of about 70ms, while smaller responses were roughly neutral. For brand, quality, extraction, and classification-style endpoints, gzip is still supported as an option. The benefit depends on how large the response is: small JSON payloads may not move much, while larger AI or crawl-derived responses can benefit from the same transfer-size reduction. ## Enable gzip Most HTTP clients already request compressed responses and decompress them automatically. The main rules are: * Don't override `Accept-Encoding` with `identity` unless you are deliberately measuring an uncompressed baseline. * Let Ruby `Net::HTTP` and Go `net/http` manage `Accept-Encoding`; if you set that header manually in those clients, they return compressed bytes and you must decompress them yourself. * Treat the decoded JSON as the application contract. Some clients expose `content-encoding: gzip`, some negotiate `br`, and some strip the header after decompression. All examples below read the API key from `CONTEXT_DEV_API_KEY` and request `/web/scrape/markdown` with automatic decompression. No manual `gunzip` step is needed. Run these from a backend process; never expose the secret API key in browser code. ```bash cURL theme={null} curl --compressed -G https://api.context.dev/v1/web/scrape/markdown \ -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \ --data-urlencode "url=https://example.com" \ --data-urlencode "useMainContentOnly=true" ``` ```typescript TypeScript fetch theme={null} type ScrapeMarkdownResponse = { success: boolean; markdown: string; }; const apiKey = process.env.CONTEXT_DEV_API_KEY; if (!apiKey) { throw new Error("CONTEXT_DEV_API_KEY is required"); } const response = await fetch( "https://api.context.dev/v1/web/scrape/markdown?url=https%3A%2F%2Fexample.com&useMainContentOnly=true", { headers: { Authorization: `Bearer ${apiKey}`, }, }, ); if (!response.ok) { throw new Error(`Context.dev returned ${response.status}: ${await response.text()}`); } const data = (await response.json()) as ScrapeMarkdownResponse; console.log(data.markdown); ``` ```javascript Node.js Axios theme={null} import axios from "axios"; const apiKey = process.env.CONTEXT_DEV_API_KEY; if (!apiKey) { throw new Error("CONTEXT_DEV_API_KEY is required"); } const { data } = await axios.get( "https://api.context.dev/v1/web/scrape/markdown", { params: { url: "https://example.com", useMainContentOnly: true, }, headers: { Authorization: `Bearer ${apiKey}`, }, }, ); console.log(data.markdown); ``` ```python Python theme={null} import os import requests api_key = os.environ["CONTEXT_DEV_API_KEY"] response = requests.get( "https://api.context.dev/v1/web/scrape/markdown", params={"url": "https://example.com", "useMainContentOnly": "true"}, headers={"Authorization": f"Bearer {api_key}"}, timeout=60, ) response.raise_for_status() data = response.json() print(data["markdown"]) ``` ```ruby Ruby Net::HTTP theme={null} require "json" require "net/http" require "uri" api_key = ENV.fetch("CONTEXT_DEV_API_KEY") uri = URI("https://api.context.dev/v1/web/scrape/markdown") uri.query = URI.encode_www_form( url: "https://example.com", useMainContentOnly: "true", ) request = Net::HTTP::Get.new(uri) request["Authorization"] = "Bearer #{api_key}" response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http| http.request(request) end unless response.is_a?(Net::HTTPSuccess) raise "Context.dev returned #{response.code}: #{response.body}" end data = JSON.parse(response.body) puts data.fetch("markdown") ``` ```go Go net/http theme={null} package main import ( "encoding/json" "fmt" "net/http" "net/url" "os" "time" ) type ScrapeMarkdownResponse struct { Success bool `json:"success"` Markdown string `json:"markdown"` } func main() { apiKey := os.Getenv("CONTEXT_DEV_API_KEY") if apiKey == "" { panic("CONTEXT_DEV_API_KEY is required") } params := url.Values{} params.Set("url", "https://example.com") params.Set("useMainContentOnly", "true") req, err := http.NewRequest( http.MethodGet, "https://api.context.dev/v1/web/scrape/markdown?"+params.Encode(), nil, ) if err != nil { panic(err) } req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{Timeout: 60 * time.Second} response, err := client.Do(req) if err != nil { panic(err) } defer response.Body.Close() if response.StatusCode < 200 || response.StatusCode >= 300 { panic(fmt.Sprintf("Context.dev returned %d", response.StatusCode)) } var data ScrapeMarkdownResponse if err := json.NewDecoder(response.Body).Decode(&data); err != nil { panic(err) } fmt.Println(data.Markdown) } ``` For Node Axios and Python `requests`, install the client library first: ```bash Node.js theme={null} npm install axios ``` ```bash Python theme={null} pip install requests ``` With these clients, the decoded JSON is what your application sees. Compression and decompression happen at the HTTP layer. ## Verify compression To confirm gzip is being negotiated, inspect the response headers with a client that does not hide the wire-level response: ```bash theme={null} curl -s -o /dev/null -D - -G https://api.context.dev/v1/web/scrape/markdown \ -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \ -H "Accept-Encoding: gzip" \ --data-urlencode "url=https://example.com" ``` Look for: ```text theme={null} content-encoding: gzip ``` If you see no `content-encoding` header in application code, the response may be small enough that compression is not worth applying, or your client may have already decompressed it before exposing headers. That is normal for clients like Axios, Ruby `Net::HTTP`, and Go `net/http`. Node's built-in `fetch` may expose `gzip` or `br` depending on what it negotiated; the response body is still decoded before `response.json()` runs. ## Where compression helps most Gzip improves transfer size for any compressible JSON response, but the latency impact depends on payload size and network conditions. | Response shape | Expected impact | | -------------------------------------- | ------------------------------------------------------------------------------------------------------- | | Large rendered HTML | Highest benefit. HTML compresses well, and `/web/scrape/html` can return hundreds of KB or multiple MB. | | Full-site crawl responses | High benefit. `/web/crawl` can return many Markdown documents in one JSON response. | | Markdown scrape responses | Moderate benefit. Markdown is usually smaller than HTML, but large pages still benefit. | | Brand and quality-style JSON responses | Optional. Use gzip when available, but expect smaller gains unless the response is large. | | Very small responses | Usually neutral. The response is already small, so transfer savings are limited. | The benchmarked scrape responses had a median compression ratio of 6.5x for HTML and 4.2x for Markdown. On slower, mobile, metered, or distant networks, the same byte reduction can translate into a larger user-visible latency win than it does from a fast server-to-server connection. ## Keep connections warm Compression reduces bytes on the wire, but connection setup can still dominate latency. Reuse HTTP connections instead of creating a fresh client for every request: ```typescript TypeScript theme={null} import axios from "axios"; import https from "node:https"; const api = axios.create({ baseURL: "https://api.context.dev/v1", headers: { Authorization: `Bearer ${process.env.CONTEXT_DEV_API_KEY}`, "Accept-Encoding": "gzip", }, httpsAgent: new https.Agent({ keepAlive: true }), }); const { data } = await api.get("/web/scrape/html", { params: { url: "https://example.com" }, }); ``` ```python Python theme={null} import os import requests session = requests.Session() session.headers.update({ "Authorization": f"Bearer {os.environ['CONTEXT_DEV_API_KEY']}", "Accept-Encoding": "gzip", }) response = session.get( "https://api.context.dev/v1/web/scrape/html", params={"url": "https://example.com"}, ) ``` For bulk scrape or crawl jobs, a reused client plus gzip is usually the best default: fewer transferred bytes, less network time on large responses, and less work for your application to move data around. ## Combine with payload controls Compression works best alongside endpoint parameters that avoid returning bytes you do not need: * Use `useMainContentOnly=true` when you only need the main article or page body. * Prefer `/web/scrape/markdown` over `/web/scrape/html` when plain text is enough. * Keep `includeImages=false` and `shortenBase64Images=true` for Markdown scrapes unless image references are required. * Use `includeSelectors` and `excludeSelectors` to narrow scrape output to the relevant page regions. * Use `maxAgeMs` for repeated scrapes so cached responses can be served quickly. ## Related resources Use scrape and crawl endpoints that benefit most from response compression. Production patterns for caching, timeouts, retries, and background jobs. Warm brand lookups before user-facing requests. Diagnose timeouts, retries, and slow requests.