Building low-latency semantic memory for coding agents with LanceDB

A coding agent starts each session cold. It has no idea what you decided last Tuesday, why you rejected the obvious approach, or which config value burned an afternoon. You can paste the context back in every time, or you can give the agent a memory it can query.

The second option sounds simple until you write it down as an engineering problem, because it has two constraints that pull against each other:

1. Retrieval has to be fast. Memory lookup sits inside the agent’s tool loop. If a recall call adds a noticeable stall on top of everything else the agent is doing, people stop using it.
2. Retrieval has to be token-cheap. If pulling context back in costs more tokens than just re-explaining, the memory layer is a net negative. It has to earn its place in the prompt.

This post is about how we built that layer for Kireo, and — more usefully — the things that broke along the way. Honest framing up front: this isn’t a scale-bragging post. It runs on one small VM, and the interesting part is how far a modest box goes when the retrieval path stays lean.

The token objection, head-on

The most common pushback on any agent-memory idea is: isn’t this just going to bloat every prompt? Fair worry, and the answer is entirely in the design.

Memory is an MCP tool the agent calls on demand, not a blob injected into every turn. When the agent needs prior context it calls memory_search, gets back a small ranked set of hits (default 10, hard cap 50), and spends tokens only on those. Nothing is prepended to the system prompt; nothing runs on turns where the agent doesn’t ask.

That distinction — pull, not push — is the whole reason the token math works. A per-turn injection scheme pays for memory on every message whether it helps or not. An on-demand top-k tool pays only when the model judges the context worth retrieving, and the budget is bounded by limit.

The server ships eight tools over MCP stdio — memory_save, memory_search, memory_recall, memory_get, memory_update, memory_delete, memory_list_namespaces, and memory_health — and every MCP client (Claude Code, Cursor, and others) sees the same set. But memory_search is the one that has to be both fast and frugal, so that’s where the engineering went.

The stack: small on purpose

The whole retrieval path:

• LanceDB — an embedded columnar vector store. No separate database server; it’s a library that reads Lance-format files off a disk volume mounted into the API and worker containers. Memory content lives here.
• Neon Postgres — holds metadata only (row bookkeeping, embedding status). It never stores the memory body. That split matters later.
• Self-hosted embeddings — HuggingFace’s Text Embeddings Inference (TEI) running intfloat/multilingual-e5-small, a 384-dimension model, on CPU. It speaks an OpenAI-compatible /v1/embeddings endpoint on the internal network.

Two of those choices are the load-bearing ones.

LanceDB is embedded, not a service. For a workload this size, a managed vector database is overkill. LanceDB is columnar on disk, the query path is a library call, no network hop. The vector column is an Arrow FixedSizeList(dim) — remember that, it comes back to bite us.

384 dimensions on CPU is enough, and it’s cheap. Memory snippets are short — a decision, a gotcha, a config note. You don’t need a 1536- or 3072-dim frontier model to separate “we chose Postgres row-level security over app-layer checks” from “the CI cache key needs the lockfile hash.” Smaller vectors mean cheaper ANN and less storage, and a small e5 model does short-text semantic matching well on CPU. The one catch: e5-family models want asymmetric prefixes — passage: for stored documents, query: for search queries — supplied from config and empty for OpenAI-style models. (That innocuous trailing space caused a real bug; more below.)

A side effect of TEI’s OpenAI-compatible endpoint: the same client code talks to the self-hosted model by pointing a base URL at the internal service — swapping providers was a config change, not a rewrite. The cache and rate-limit layers use the same trick: a self-hosted Redis fronted by a shim speaking the Upstash REST API, so the REST client needs no managed account.

Hybrid search, with a graceful-degradation ladder

Pure vector search misses exact-match cases (a specific error code, a function name). Pure keyword search misses paraphrase. So memory_search runs both and fuses them.

The flow:

1. Embed the query (cached, so repeated searches don’t re-embed).
2. In parallel, run an ANN search over the vector column (top 50) and a native full-text search (top 50), both scoped to the caller’s tenant.
3. Fuse the two ranked lists with reciprocal rank fusion and slice to the requested limit.

RRF is deliberately boring: it combines rankings by 1/(k + rank) without needing the two scoring systems to share a scale — cosine distance and BM25 don’t. Robust and cheap, which is what you want in the hot path.

The interesting part is the degradation ladder: in a lean stack any dependency can be briefly unavailable, and search still has to return something:

• Query embedding fails (provider hiccup, timeout)? Log a degraded event so it alerts, drop the vector arm, search keyword-only.
• No FTS index (common in a fresh dev environment)? Fall back to a bounded case-insensitive substring scan ranked by importance.
• Both arms empty? Return an empty result, not a 500.

None of that is glamorous, but it’s the difference between “memory occasionally returns fewer hits” and “memory throws inside the agent’s tool loop.”

Namespaces and tenant isolation

Every row carries a user_id and a namespace (e.g. code-my-app for an indexed repo). Isolation is a user_id predicate pushed into every LanceDB query, plus a belt-and-suspenders assertion after results return: if any row’s user_id doesn’t match the caller, the code throws instead of leaking it. The filter should never be wrong — so we check anyway, on every read path. Cross-tenant leakage is the one bug you never want to ship, and a three-line assertion is cheap insurance.

War story 1: the embedding-dimension migration

The stack didn’t start at 384 dimensions. It started at 1536 against a hosted model; moving to the self-hosted 384-dim e5 model meant every stored vector was now the wrong length.

Here’s where LanceDB’s FixedSizeList(dim) schema stops being an implementation detail. The vector column’s dimension is baked into the table schema, and in the version we run there is no in-place column resize and no table rename. Once the dimension changes, every insert fails against the old table, and you can’t quietly widen the column.

The migration is dump-drop-recreate:

1. Read every row out of the memories table.
2. Write a durable JSON dump to disk before anything destructive — memory content lives only in LanceDB (Postgres has metadata only), so if the recreate dies halfway, that dump is the only copy. Old vectors are dropped on reload anyway, so the dump excludes them.
3. Drop the table, recreate it empty at the new dimension.
4. Re-insert every row in batches with embedding = null and embedding_status = 'queued', then reset the Postgres status rows to queued too so the backfill job re-embeds everything with the new model.

The migration is idempotent — if the table is already at the target dimension it’s a no-op — which matters when you’re running it by hand on a live box, unsure whether the last attempt finished.

And there’s a subtle second-order bug this exposed. The embedding cache is keyed by model + content hash — but not by dimension. So after a same-model endpoint change that alters the vector length, a stale cached vector of the old length would sail past the cache lookup and get written into the freshly recreated table, breaking the insert. The fix is a dimension guard at two layers: the embed client asserts vector.length === EMBEDDING_DIM before returning (a wrong-length embed fails and degrades to the queue instead of corrupting the table), and the cache read treats a length mismatch as a miss. The rule: never let a wrong-dimension vector reach the table, enforced at every point one could enter.

War story 2: making batch writes idempotent

Indexing a repo uploads symbols in batches — up to 100 per request. Batches time out sometimes, and the obvious retry (re-send the batch) creates duplicates unless the write path is idempotent.

The fix is content-hash dedup as a single set query, not N point lookups. Before inserting a batch, one query fetches the active (non-deleted) rows whose content_hash is in the batch’s hashes — content_hash IN (...) for the whole batch — and skips them. Re-running the same upload is safe: identical content hashes to identical rows, retries don’t multiply.

Two things I like here. First, one IN query keeps dedup off the per-item hot path — one query per batch, not one per symbol. Second, it’s the same guarantee surfaced in the CLI docs: if a batch upload times out, re-running the same command is safe. Not an internal nicety but a documented contract — the person hitting the timeout is the one who needs to trust the retry.

War story 3: what “trash” actually means

Delete is where naive implementations quietly lose data or lie about counts. Our fixes here were all, at heart, about semantics.

Deletes are soft by default: a delete stamps deleted_at and sets a 30-day expires_at restore window. The row stays in the table, filtered out of normal reads by deleted_at IS NULL. Restore checks the window and refuses if it’s expired; a TTL sweep physically removes rows past expires_at.

That created three follow-on requirements that each needed explicit handling:

• A trash view needs the inverse filter. Listing deleted items isn’t “include deleted” — it’s “only deleted.” Different queries, and “only deleted” has to win when both flags are set. Get it subtly wrong and the user sees an empty or wrong trash.
• Counts have to match the view. The “trashed” count is its own query, not total − active, because off-by-a-little count math is exactly what users notice and stop trusting.
• Updates are in-place, never delete-then-add. LanceDB will let you delete and re-add a row, but a crash between those calls permanently loses it while its Postgres metadata survives — a torn write. Every mutation (edit, soft-delete, restore, namespace rename) is an in-place update, so there’s no window where the row doesn’t exist.

One more LanceDB-shaped wrinkle: a plain scan has no ORDER BY. To list newest-first without materializing a million rows, the list path streams every matching batch, re-sorting as they arrive and truncating to limit + 1 so only the current top page stays in memory. It’s more code than ORDER BY ... LIMIT, but it’s what the storage engine actually supports.

The unglamorous infrastructure footnotes

Three more that belong in any honest “one VM” story:

• A single log file once grew to 62 GB. A worker got stuck in an error loop against an exhausted upstream and its JSON logs ate the disk. The fix is boring and permanent: every container caps log rotation at 10 MB × 3 files. On a small box, an unbounded log is a time bomb.
• The embedding image is pinned by digest. An earlier CPU tag of TEI bundled a dependency version that failed model download with a cryptic “relative URL without a base.” It’s now pinned to a known-good digest, with a comment saying exactly why, so an innocent :latest pull can’t resurrect the bug.
• Config whitespace is load-bearing. Docker’s dotenv parsing trims trailing whitespace, which silently ate the trailing space in the e5 passage: prefix and quietly degraded recall until the value was quoted. The kind of bug that has no stack trace.

None of these are clever. They’re the tax for running a real workload on modest hardware — and writing them down means the next person (often me, three months later) skips the debugging.

Where this runs

This is the memory layer behind Kireo, an MCP server that gives Claude Code, Cursor, and any MCP client one shared long-term memory — save decisions and gotchas as you work, recall them from any tool, and browse, edit, or delete everything in a web dashboard, with full JSON export if you ever want to walk away with your data. It’s in free beta right now with generous limits and no card required.

Install is one line:

claude mcp add kireo --scope user --env KIREO_API_KEY=ki_sk_xxx -- npx -y --package=@kireo/mcp-server kireo-mcp

Enter fullscreen mode Exit fullscreen mode

Grab a key and read the docs at kireo.app. On privacy: the server only sends what you explicitly pass to memory_save; it never reads your code, and code indexing stores derived embeddings and file paths, not your source — see the data-storage policy. The MCP server is on npm and GitHub.

I’ve kept latency claims out on purpose — I’d rather ship measured numbers than round ones, and a follow-up profiling the search path is on the list. If you build agent memory on a small stack and hit a different set of walls, I’d like to hear which ones.