Build a Semantic Search Engine for a Website

The hard part of site search is not the search. It is the second run: five thousand pages, of which twenty changed, and an index job that must not spend an hour and a bill re-embedding the other four thousand nine hundred and eighty. Content hashing per chunk turns that into a job that finishes in seconds.

Four jobs, not one script

Separate them, because they fail differently and run on different schedules.

Job Description crawl Fetch URLs, store raw HTML and an ETag. Network-bound, fails on individual pages. extract HTML to clean text plus title. Pure function of the HTML — cheap to re-run. index Chunk, hash, embed only what is new. Costs money. The subject of this page. serve Query embedding plus search. Latency-sensitive, must never block on the others.

Keeping raw HTML is the decision people regret skipping. When you change the extractor — and you will, the first time a template change puts the cookie banner in every chunk — you re-extract from disk instead of re-crawling the site.

Decide early what “a page” means for your site, because it is not obvious and it changes the results. A paginated article is one document split across five URLs; a documentation page with anchors is arguably five documents at one URL; a listing page is a page you should not index at all, because it contains fragments of a hundred others and will match every query weakly. Write down the inclusion rule — a URL pattern list is enough — and keep it next to the crawler, or it will live only in whoever wrote it.

Crawling politely

Your own site still deserves a conditional request and a delay. The standard library has both.

# crawl.py
import time, urllib.request, urllib.robotparser
from urllib.parse import urljoin, urlparse

UA = "yoursite-search/1.0 (+https://yoursite.example/about-our-crawler)"
DELAY = 0.5

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://yoursite.example/robots.txt")
rp.read()

def fetch(url, etag=None):
    if not rp.can_fetch(UA, url):
        return None, None, "disallowed"
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    if etag:
        req.add_header("If-None-Match", etag)
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            return r.read().decode("utf-8", "replace"), r.headers.get("ETag"), "ok"
    except urllib.error.HTTPError as e:
        if e.code == 304:
            return None, etag, "unchanged"
        return None, None, "http-" + str(e.code)
    finally:
        time.sleep(DELAY)

Enter fullscreen mode Exit fullscreen mode

A 304 Not Modified is the cheapest possible answer and most static site hosts send ETags without being asked. On a well-behaved site this alone cuts a nightly crawl to a handful of real downloads — before any of the embedding savings below.

Getting the content out of the HTML

The goal is the article, not the chrome. Navigation, footer and cookie banner repeat on every page, and if they end up in your chunks every query matches every page a little bit, which is the classic symptom of a search engine that returns plausible nonsense.

1. Prefer a container the template already gives you: <main>, <article>, or a known content class. On your own site you know this — do not use a generic readability heuristic when a CSS selector is exact.
2. Strip <script>, <style>, <nav>, <header>, <footer> and anything aria-hidden.
3. Keep <h1>–<h3> as a heading path per chunk, the way the PDF build does. It is the same trick and it works for the same reason.
4. Assert on the result: if extracted text is under 200 characters for a page you know is long, the selector broke. Fail the job loudly rather than indexing empty pages.

Python’s stdlib html.parser is enough for this and it has no dependencies; content extraction from HTML gets harder the less control you have over the templates.

The re-index that does not re-embed

Here is the mechanism, and it is only a few lines. Hash each chunk; keep the hash next to the vector; on re-index, embed only chunks whose hash is absent from the table, and delete rows whose hash no longer appears for that URL.

# index.py
import hashlib, json, sqlite3

DB = sqlite3.connect("search.db")
DB.executescript("""
CREATE TABLE IF NOT EXISTS chunk (
  hash  TEXT PRIMARY KEY,        -- sha256 of the normalised chunk text
  url   TEXT NOT NULL,
  ord   INTEGER NOT NULL,
  title TEXT NOT NULL,
  text  TEXT NOT NULL,
  vec   TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS chunk_url ON chunk(url);
""")

def chunk_hash(text):
    return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()

def reindex(url, title, pieces):
    wanted = {chunk_hash(p): (i, p) for i, p in enumerate(pieces)}
    have = {h for (h,) in DB.execute(
        "SELECT hash FROM chunk WHERE url = ?", (url,))}

    stale = have - wanted.keys()
    if stale:
        DB.executemany("DELETE FROM chunk WHERE hash = ?",
                       [(h,) for h in stale])

    fresh = [h for h in wanted if h not in have]
    if fresh:
        vecs = embed([wanted[h][1] for h in fresh])       # the only paid call
        DB.executemany(
            "INSERT OR REPLACE INTO chunk (hash,url,ord,title,text,vec) "
            "VALUES (?,?,?,?,?,?)",
            [(h, url, wanted[h][0], title, wanted[h][1], json.dumps(v))
             for h, v in zip(fresh, vecs)],
        )
    DB.commit()
    return len(fresh), len(stale)

Enter fullscreen mode Exit fullscreen mode

Two details do the work. The hash is over " ".join(text.split()), so a reflowed paragraph or a changed indent does not count as a change — otherwise a template tweak re-embeds the site. And ord is stored but not hashed, so moving a paragraph up the page reorders rows without re-embedding anything.

The failure mode to design against is the opposite one: a chunk whose text is identical on two URLs shares a hash, and with hash as the primary key the second URL overwrites the first. If your site has genuinely duplicated boilerplate paragraphs, make the key sha256(url + text) instead and accept that you re-embed duplicates once per URL.

Nightly job over 5,000 pages, ~6 chunks each = 30,000 chunks.

Full re-embed:        30,000 x 300 tokens = 9.0M tokens
Incremental, 20 pages changed:
                         120 x 300 tokens = 36,000 tokens
                                          = 0.4% of the full run.

At an embedding price of $0.02 per million tokens that is $0.18 versus
$0.0007 — small either way. The reason to do it is the clock: a full run
is 30,000 embeddings' worth of round trips and rate limits, and an
incremental one finishes before you have made coffee.

Enter fullscreen mode Exit fullscreen mode

Serving the query

One embedding call for the query, then the same dot product as any other vector search — but two things belong in the serving path that do not belong in a batch script.

• Cache query embeddings. Site search queries follow a brutal power law; a dictionary keyed on the normalised query, capped at a few thousand entries, removes a network round trip from most searches.
• Group results by URL. Users want pages, not chunks. Take the best-scoring chunk per URL, show its text as the snippet, and never return the same page twice.
• Have a fallback. If the embedding call fails or times out, fall through to keyword search rather than showing an error. Search that degrades is search that is up.

Why pure semantic search disappoints

The first complaint will be that searching for an exact term — a product code, a person’s surname, a version number — returns pages about the general topic instead of the page containing the term. That is embeddings working as designed: they encode meaning, and a serial number has no meaning to encode.

The fix is not a better model. Run keyword search alongside — SQLite’s FTS5 gives you BM25 in the database you already have — and merge the two result lists. Reciprocal rank fusion is the standard merge and it is three lines: score each document as the sum over lists of 1 / (60 + rank), then sort. Hybrid retrieval beats either half on nearly every real corpus, and the two methods fail on complementary queries, which is exactly why merging works.

The latency budget for a search box

Site search is judged against the browser’s own autocomplete, which is instant, so the budget is tight and the terms are worth writing down before you discover them.

Target: results rendered within 300 ms of the user stopping typing.

  debounce                       150 ms   <- your choice, and it is half the budget
  request to your server          20 ms
  query embedding call            80-250 ms  <- a NETWORK CALL, and the risk
  vector scan, 30,000 chunks       15 ms   (NumPy: one 30,000 x 1,536 matmul)
  group, snippet, serialise         5 ms
  render                           15 ms

Two things follow.

1. The embedding call is the only term you do not control, and it is
   comparable in size to the entire rest of the budget. Cache it, and
   consider a smaller embedding model for QUERIES than for documents only
   if the two are the same model — they must be, or the vectors are not
   comparable.

2. Debouncing at 150 ms rather than 300 ms is a free 150 ms, at the cost
   of more requests for people who type in bursts. Since the cache absorbs
   repeated prefixes, that cost is smaller than it looks.

Enter fullscreen mode Exit fullscreen mode

The pure-Python scan from the RAG build is 1–3 seconds at this corpus size and does not fit in this budget at all; the same arithmetic in NumPy as a single matrix multiply does, comfortably. That is the actual reason to reach for a numeric library here, and it is worth being precise about — not “NumPy is faster”, but “the budget is 300 ms and one term of it is 1,000”.

If the embedding call is unavailable or slow, return the keyword results and mark them as such rather than showing a spinner. A search box that is sometimes merely good is better than one that is sometimes absent, and degrading rather than failing is the pattern that keeps a feature enabled through its first bad week.

Running it

1. Crawl and index on a schedule that matches how often the site changes. Nightly is right for most; a docs site that ships hourly wants a webhook from the deploy instead.
2. Write the index to a new SQLite file and rename it over the old one when the job succeeds. An atomic rename means search is never served from a half-built index.
3. Log queries with zero results above a score threshold. That log is the highest-value piece of content research your site will produce: it is a list, in your visitors’ own words, of things they expected you to have written and you have not. Read it weekly. The second most valuable log is queries where somebody searched again within ten seconds, which is the signature of results that looked wrong at a glance.
4. When you change embedding model, everything must be re-embedded — vectors from two models are not comparable. Plan that migration as a dual-write rather than a big-bang swap.

Related

• Build a RAG Pipeline From Scratch Without a Framework
• Build a Recommendation Feed With Embeddings