Build RAG from websites

Use website content as a source for retrieval-augmented generation (RAG): retrieve relevant passages, then pass them to a model with the user’s question. Context.dev renders the pages and returns Markdown; your retrieval stack handles chunking, embeddings, search, and answers. SiteGPT uses website scraping to power support chatbots. For questions that need fresh pages outside a fixed corpus, use live agent web tools alongside or instead of a persistent index. You’ll need an API key and your preferred SDK from the Quickstart, an embedding model, and a vector index. The ingestion pipeline below uses TypeScript; the API request is shown in every supported SDK. Crawling costs 1 credit per successfully scraped page, plus any PDF OCR charges.

How it works

Crawl the source pages

Start with a bounded page count and path scope. Inspect the result before widening it.

curl https://api.context.dev/v1/web/crawl \
  --request POST \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
  "url": "https://docs.example.com/product/",
  "urlRegex": "^https://docs\\.example\\.com/product/",
  "maxPages": 100,
  "maxDepth": 4,
  "useMainContentOnly": true,
  "includeLinks": true
}'

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.web.webCrawlMd({
  url: "https://docs.example.com/product/",
  urlRegex: "^https://docs\\.example\\.com/product/",
  maxPages: 100,
  maxDepth: 4,
  useMainContentOnly: true,
  includeLinks: true,
});

console.log(response);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.web.web_crawl_md(
    url="https://docs.example.com/product/",
    url_regex="^https://docs\\.example\\.com/product/",
    max_pages=100,
    max_depth=4,
    use_main_content_only=True,
    include_links=True,
)

print(response)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.web.web_crawl_md(
  url: "https://docs.example.com/product/",
  url_regex: "^https://docs\\.example\\.com/product/",
  max_pages: 100,
  max_depth: 4,
  use_main_content_only: true,
  include_links: true,
)

pp response

package main

import (
    "context"
    "fmt"
    "os"

    contextdev "github.com/context-dot-dev/context-go-sdk/v2"
    "github.com/context-dot-dev/context-go-sdk/v2/option"
    "github.com/context-dot-dev/context-go-sdk/v2/packages/param"
)

func main() {
    client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))
    response, err := client.Web.WebCrawlMd(context.Background(), contextdev.WebWebCrawlMdParams{
        URL:                "https://docs.example.com/product/",
        URLRegex:           param.NewOpt("^https://docs\\.example\\.com/product/"),
        MaxPages:           param.NewOpt(int64(100)),
        MaxDepth:           param.NewOpt(int64(4)),
        UseMainContentOnly: param.NewOpt(true),
        IncludeLinks:       param.NewOpt(true),
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(response)
}

<?php

require __DIR__.'/vendor/autoload.php';

$client = new ContextDev\Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->web->webCrawlMd(
    url: "https://docs.example.com/product/",
    urlRegex: "^https://docs\\.example\\.com/product/",
    maxPages: 100,
    maxDepth: 4,
    useMainContentOnly: true,
    includeLinks: true,
);

print_r($response);

TypeScript ingestion

Filter unsuccessful or empty pages before indexing:

import ContextDev from "context.dev";

const client = new ContextDev({
  apiKey: process.env.CONTEXT_DEV_API_KEY,
});

const crawl = await client.web.webCrawlMd({
  url: "https://docs.example.com/product/",
  urlRegex: "^https://docs\\.example\\.com/product/",
  maxPages: 100,
  maxDepth: 4,
  useMainContentOnly: true,
  includeLinks: true,
});

const pages = crawl.results.filter(
  (page) => page.metadata.success && page.markdown.trim().length > 0,
);

console.log({
  indexedCandidates: pages.length,
  failed: crawl.metadata.numFailed,
  skipped: crawl.metadata.numSkipped,
});

The crawl can return partial progress when it reaches its time budget. numFailed and numSkipped are not the same as an endpoint-level failure. Decide whether your ingestion job can accept a partial corpus or should retry missing URLs. followSubdomains defaults to false. Treat documentation hosts and other subdomains as explicit source decisions; keep the urlRegex scope consistent if you enable them. The crawler treats www and the apex domain as equivalent, but your index should still normalize the URLs it accepts.

Normalize page identity before chunking

Use one canonical key for a page before generating chunk IDs. Drop fragments and known tracking parameters, but preserve query parameters that change the document, language, or version. Only merge trailing-slash variants, redirects, or canonical-link targets when you have verified they identify the same content.

page-identity.ts

export function canonicalSourceUrl(raw: string) {
  const url = new URL(raw);
  if (!["https:", "http:"].includes(url.protocol)) throw new Error("Expected a web URL");
  url.hash = "";
  for (const name of [...url.searchParams.keys()]) {
    if (name.startsWith("utm_") || name === "gclid" || name === "fbclid") {
      url.searchParams.delete(name);
    }
  }
  return url.href;
}

const uniquePages = new Map<string, (typeof pages)[number]>();
for (const page of pages) {
  const key = canonicalSourceUrl(page.metadata.url);
  const existing = uniquePages.get(key);
  if (existing && existing.markdown !== page.markdown) {
    throw new Error("Review conflicting content for one canonical URL");
  }
  if (!existing) uniquePages.set(key, page);
}

Keep the original retrieved URL, page title, and normalized key in your source manifest. Recheck redirected or canonicalized URLs against the allowed corpus before indexing. If two entries share a key but disagree on content, resolve that conflict instead of silently combining them. Preserve separate titles and source records for mirrored pages when their provenance matters.

A path pattern is an ingestion boundary, not an authorization layer. Enforce tenant and document access again when writing to and querying your vector index.

Chunk with source metadata

Split on semantic boundaries first, then enforce your embedding model’s token limit. A useful record contains enough metadata to cite, replace, and delete the chunk later.

import { createHash } from "node:crypto";

type Chunk = {
  id: string;
  sourceUrl: string;
  pageTitle: string;
  heading: string;
  text: string;
  contentHash: string;
  crawledAt: string;
};

function splitByHeading(markdown: string) {
  const sections: Array<{ heading: string; text: string }> = [];
  let heading = "Introduction";
  let lines: string[] = [];
  let fence: { marker: string; length: number } | null = null;

  const flush = () => {
    const text = lines.join("\n").trim();
    if (text) sections.push({ heading, text });
    lines = [];
  };

  for (const line of markdown.split("\n")) {
    const delimiter = line.match(/^\s{0,3}(`{3,}|~{3,})(.*)$/);
    if (delimiter) {
      const marker = delimiter[1][0];
      if (!fence) {
        fence = { marker, length: delimiter[1].length };
      } else if (
        marker === fence.marker &&
        delimiter[1].length >= fence.length &&
        delimiter[2].trim() === ""
      ) {
        fence = null;
      }
      lines.push(line);
      continue;
    }
    const match = line.match(/^#{1,3}\s+(.+)$/);
    if (match && !fence) {
      flush();
      heading = match[1];
    } else {
      lines.push(line);
    }
  }
  flush();
  return sections;
}

function makeChunk(
  sourceUrl: string,
  pageTitle: string,
  heading: string,
  text: string,
): Chunk {
  const contentHash = createHash("sha256").update(text).digest("hex");
  const id = createHash("sha256")
    .update(`${sourceUrl}\n${heading}\n${contentHash}`)
    .digest("hex");

  return {
    id,
    sourceUrl,
    pageTitle,
    heading,
    text,
    contentHash,
    crawledAt: new Date().toISOString(),
  };
}

The example splits only by headings. Add a tokenizer-aware sub-split for long sections and small overlap where the text requires continuity. Avoid blind fixed-width chunks that separate headings, code, tables, or list context from their content.

Embed and index

Use the same embedding model and dimensions for ingestion and retrieval. Batch requests within your provider’s limits.

const chunks = [...uniquePages.entries()].flatMap(([sourceUrl, page]) =>
  splitByHeading(page.markdown).map(({ heading, text }) =>
    makeChunk(sourceUrl, page.metadata.title || sourceUrl, heading, text),
  ),
);

for (const batch of batchesOf(chunks, 100)) {
  const vectors = await embeddingProvider.embedMany(batch.map((chunk) => chunk.text));

  await vectorIndex.upsert(
    batch.map((chunk, index) => ({
      id: chunk.id,
      values: vectors[index],
      metadata: chunk,
    })),
  );
}

batchesOf, embeddingProvider, and vectorIndex represent your chosen libraries. Store the embedding-model version with the index so a model migration becomes an explicit reindex operation.

Retrieve with citations

At query time:

Apply the caller’s access-control filter.
Embed the question with the same model used for the corpus.
Retrieve a small candidate set.
Optionally rerank it.
Give the answer model the chunk text, heading, and source URL.
Require citations to come only from the supplied URLs.
Return “I don’t have enough evidence” when retrieval is weak.

Do not instruct the model to answer from general knowledge when the product promise is source-grounded. Treat content found in crawled pages as untrusted data, not as instructions to the model.

Refresh and delete stale content

Upserts add new or changed chunks but do not remove deleted pages or sections. Check crawl coverage before treating missing content as deleted: a failed or skipped page may still exist. On a complete, validated refresh:

record the set of source URLs and chunk IDs found in the new crawl;
upsert new and changed chunks;
delete old IDs that are no longer present;
keep the previous successful index active until the new run passes checks;
swap versions atomically when possible.

For frequently changing sources, use website monitors to trigger targeted refreshes. Keep a periodic full reconciliation to catch missed events and deletions. Keep a per-source manifest containing the last successful content hash, active chunk IDs, last successful retrieval time, latest attempt status, and index version. For an incremental refresh:

Observation	Index action
Successful page with unchanged content hash	Update the observation time and skip embedding.
Successful changed page	Build and validate replacement chunks, then replace that page’s old chunk set.
Failed, blocked, skipped, or unexpectedly empty page	Retain the last good chunks and mark the source stale.
Missing from a crawl that hit a page or time limit	Leave the source unchanged until coverage is resolved.
Confirmed removal or an explicit source deletion	Remove the page and its chunks through your deletion policy.

Stage replacement chunks under a new version so a failed embedding batch cannot leave a page half-updated. Keep tenant and document access filters attached to both versions. An upsert of new chunk IDs alone is insufficient: obsolete IDs must be retired after the new version is ready.

Evaluate answers and coverage

Create a small, representative question set before launch and track:

Layer	Useful measure
Crawl	Expected URL coverage, failed pages, and empty pages
Chunking	Sections split incorrectly or missing necessary context
Retrieval	Recall of a relevant source in the top results
Answer	Correctness, citation support, and refusal when evidence is absent
Freshness	Time from source change to searchable update
Access	Cross-tenant or unauthorized retrieval attempts blocked

Re-run the evaluation when you change the crawl scope, chunker, embedding model, index settings, reranker, or prompt.

Crawl a website

Collect linked pages for your content pipeline.

Discover website URLs

Find URLs before setting your crawl scope.

Zero data retention

Understand retention controls and eligibility requirements.

Brand experiences

Sales and content

Web research

Build RAG from websites

How it works

Crawl the source pages

TypeScript ingestion

Normalize page identity before chunking

Chunk with source metadata

Embed and index

Retrieve with citations

Refresh and delete stale content

Evaluate answers and coverage

Crawl a website

Discover website URLs

Zero data retention

​How it works

​Crawl the source pages

​TypeScript ingestion

​Normalize page identity before chunking

​Chunk with source metadata

​Embed and index

​Retrieve with citations

​Refresh and delete stale content

​Evaluate answers and coverage

Crawl a website

Discover website URLs

Zero data retention

How it works

Crawl the source pages

TypeScript ingestion

Normalize page identity before chunking

Chunk with source metadata

Embed and index

Retrieve with citations

Refresh and delete stale content

Evaluate answers and coverage