RAG Retrieval Accuracy: 38%. After the Fix: 87%. The Model Was Never Touched.

That’s a rebuild I shipped. The system: a RAG assistant for fraud analysts — ask it “how do we handle card testing followed by a successful auth?” and it should answer from the team’s own SOPs and case history. The complaint: the answers were wrong, therefore the model must be dumb, therefore procurement should buy a bigger model.

The model was fine. It was answering perfectly — from garbage context. Walk the forensic trail with me, because every step is checkable on your own system this week.

Exhibit A: the chunking was destroying meaning before anything was embedded

The ingestion split SOP documents every N characters, mid-sentence. Which means half the vectors in the index encoded fragments like this:

chunk_147 = "...ing to a freight forwarder. In these cases, do NOT"
chunk_148 = "cancel the order immediately. First verify the customer via"

Enter fullscreen mode Exit fullscreen mode

The policy — don’t cancel, verify first — exists in no single chunk. An embedding can’t encode a meaning that isn’t in its input. Retrieval was being asked to find semantics the pipeline had already shredded.

Fix one: chunk on structure (sections, paragraphs), never on character counts, with enough overlap that no rule straddles a boundary.

Exhibit B: dense-only retrieval, bimodal queries

Fraud analyst queries split into two populations: pattern questions (“high-value order, new account, rushed shipping”) and identifier questions (“what’s the SOP for decline code 4863?”, “rule VEL-013 rationale”). The system was dense-only — and embeddings treat a rare token like 4863 as noise, so identifier queries retrieved similar-feeling chunks instead of the literal match. Half the query population was structurally doomed regardless of model quality.

Fix two: hybrid retrieval — BM25 for the identifiers, embeddings for the patterns, reciprocal rank fusion to merge.

Exhibit C: nobody could see any of this, because quality was a rumor

No golden dataset. No retrieval metric. The system’s accuracy was whatever the loudest anecdote said it was. So fix three came first in reality: build the eval before touching the pipeline, or you’re tuning blind.

# The eval harness that made the whole rebuild measurable.
# Golden set: real analyst questions + the chunk IDs that actually answer them.
golden = [
    {"q": "SOP for decline code 4863 repeated then success", "relevant": {"sop_12_3"}},
    {"q": "customer disputes delivered order with proof",     "relevant": {"sop_07_1", "case_1177"}},
    {"q": "rule VEL-013 rationale and exceptions",            "relevant": {"rule_vel_013"}},
    {"q": "new account high value order mismatched device",   "relevant": {"case_1203", "sop_04_2"}},
    # ... ~150 more, sampled from real analyst search logs
]

def recall_at_k(retrieve_fn, golden, k=5) -> float:
    hits = 0
    for item in golden:
        retrieved_ids = {doc_id for doc_id, _ in retrieve_fn(item["q"], k)}
        if retrieved_ids & item["relevant"]:
            hits += 1
    return hits / len(golden)

for name, fn in [("dense-only, bad chunks", retrieve_v1),
                 ("dense-only, clean chunks", retrieve_v2),
                 ("hybrid, clean chunks", retrieve_v3)]:
    print(f"{name:28s} recall@5 = {recall_at_k(fn, golden):.0%}")

Enter fullscreen mode Exit fullscreen mode

dense-only, bad chunks       recall@5 = 38%
dense-only, clean chunks     recall@5 = 61%
hybrid, clean chunks         recall@5 = 87%

Enter fullscreen mode Exit fullscreen mode

That table is the whole story. Chunking alone bought 23 points. Hybrid bought another 26. The model — the thing everyone wanted to replace — appears nowhere in it, because generation is downstream of retrieval and the model can only be as right as its context.

Note where the golden set came from: real analyst search logs, not invented test queries. Invented queries skew toward the pattern style (that’s how engineers imagine search), which is exactly how the identifier failure mode stays invisible. Your eval set inherits the biases of whoever writes it — sample from production or measure a fiction.

Why fraud teams should care more than most

Because the stakes of a wrong answer are asymmetric and immediate. An assistant that retrieves the wrong SOP doesn’t produce a mildly worse paragraph — it produces an analyst releasing an order that matches a known mule-address pattern, or holding a legitimate gift purchase and burning a good customer. Retrieval quality in this domain is decision quality, one hop removed.

The principle: before you blame the model, audit what you fed it. RAG quality is a data pipeline problem wearing an AI costume.

When your AI gives a wrong answer — does your team blame the model first, or check the retrieval first?

I’m Vinicius Fagundes — principal data engineer and MBA lecturer in São Paulo. I build fraud and risk analytics pipelines for e-commerce through vf-insights.com.