How coding agents like Cursor quietly cut input costs by reusing KV states across turns — and what actually breaks the cache

Why my Cursor bill looked weird

I was poking around my usage dashboard in Cursor and noticed a metric I’d never paid attention to before: Cache Read.

Date            Type       Model    Tokens
May 2, 06:50 PM  Included   auto     898K   → Cache Read
May 2, 06:49 PM  Included   auto     1.4M   → Cache Write
May 2, 06:47 PM  Included   auto     346.8K → Input / Output

Enter fullscreen mode Exit fullscreen mode

Almost 99% of my input tokens for that session were served from cache. That sent me down a rabbit hole on what prompt caching actually is, and why it matters so much for agentic coding tools specifically. Here’s what I found.

What prompt caching actually is

Agentic coding tools ship a lot of context on every single call — system instructions, workspace rules, file contents, conversation history, the works. LLMs are stateless by default, so naively, all of that gets reprocessed from scratch on every request.

Prompt caching short-circuits this. It saves the intermediate computation from the transformer’s prefill stage — specifically the Key/Value states — so that when a later request shares the same leading sequence of tokens, the model reuses those states instead of recomputing them.

One important caveat: this only applies to input tokens. Output generation is still sequential, token-by-token, and gets no speedup from caching.

How the payload is actually built

Coding agents don’t just fire your question at the model. They assemble a large composite prompt, and — critically — they structure it with the most static content first and the most volatile content last:

<system_instructions> You are an expert coding assistant... </system_instructions>
<project_rules> Use functional React components... </project_rules>
<file_context_1> ... </file_context_1>
<file_context_2> ... </file_context_2>
<conversation_history> ... </conversation_history>
<user_input> Refactor this function to handle edge cases. </user_input>

Enter fullscreen mode Exit fullscreen mode

Once that sequence hits the model, the transformer tokenizes it and starts computing attention. For every token, it derives a Key (K) and a Value (V) — essentially a map of how that token relates to every other token before it. This is the expensive part: attention computation scales quadratically with sequence length, so a big context window (lots of files, long history) gets disproportionately costly to prefill.

The model doesn’t refer back to your raw text when generating a response — it refers to these computed KV matrices. That’s the artifact caching actually preserves.

Providers typically charge less for input tokens than output tokens, but agentic coding tools push such enormous input payloads that shaving cost off the input side is still very much worth doing.

Turn 1 vs. every turn after

Here’s the flow, roughly:

Turn 1 (cold start):

1. Build the prompt: system instructions + project rules + file context + your first message.
2. No cache exists yet → full prefill. The model computes KV states for every token (the expensive O(n²) pass).
3. Those KV states get written to cache storage.
4. Generation proceeds normally.

Turn 2…N (warm):

1. Build the new prompt: same system instructions + same files + prior conversation + your new message.
2. Check for a prefix cache hit against system + context.
3. Hit: read the cached KV states for the unchanged prefix, and only compute new KV states for the incremental conversation turn + your latest input.
4. Concatenate cached + new KV states, then generate.

So instead of re-running the full O(n²) prefill on every message, only the marginal new tokens get freshly processed. That’s the whole trick, and it’s why long coding sessions don’t get proportionally slower and more expensive turn after turn.

Most major providers now do this optimization implicitly under the hood. Google’s Vertex AI docs on context caching are a good primary-source reference if you want to see how one provider exposes this explicitly.

What breaks the cache

Caching depends entirely on prefix stability. The moment a token diverges from what’s cached, everything downstream of that point has to be recomputed. In practice, here’s what kills your hit rate:

• Dynamic injections — a timestamp or other changing value stuffed near the top of the prompt invalidates the prefix on every single request.
• Context reordering — shuffling the order in which files or context blocks get loaded prevents prefix matching entirely.
• Upstream file edits — editing a file that sits early in the prompt structure invalidates the cache for that file and everything that comes after it in the sequence.

Most systems rely on exact token-prefix matches, though some providers (Google’s docs linked above, for instance) expose more explicit cache handles or partial-reuse mechanisms.

The part that confused me: why appending doesn’t break the cache

This is the part that seemed contradictory at first. If attention connects every token to every other token, shouldn’t adding a new message at the end of the prompt ripple backward and change how the earlier tokens (system instructions, file context) get attended to?

It would — if the model attended in both directions. But modern generative LLMs are decoder-only and use causal (masked) self-attention: a token can only attend to tokens that came before it, never after.

That one architectural detail is what makes caching possible at all. We’re not caching a fragile, bidirectional global attention matrix — we’re caching the KV tensors of the prefix. Those K and V states for the early context are computed once and never change, no matter what you append afterward. New tokens just generate their own Queries and attend against the already-cached Keys of everything before them.

Where this goes next

Prompt caching solves the “don’t recompute what hasn’t changed” problem. It doesn’t solve the separate problem of what to put in the context window in the first place. Instead of stuffing every file into the prompt and hoping for a cache hit, coding agents increasingly rely on retrieval — RAG-style embeddings over the codebase — to selectively pull in only the relevant slices of code.

That’s next on my list to dig into: how coding tools build queryable views of a codebase with embeddings, and what that does to the cost equation.

If you want the deeper transformer-architecture background behind KV caching and attention, 3Blue1Brown’s “Large Language Models explained briefly” is a genuinely excellent, low-jargon primer.

Originally posted on Susheem’s Jottings — cross-posting here for the dev.to crowd.