BM25 is a ranking function from the 90s. No GPU, no embedding API, no per-token bill. And in a fraud operation it still wins half the queries — because fraud analysts don’t search in prose. They search in identifiers.
Watch a fraud analyst’s search history for a day: decline code 4863. Rule VEL-013. BIN 529904. Promo code FIRSTBUY20. Order #48213. These aren’t concepts. They’re strings — and “semantically similar” is the wrong answer when the user typed the literal thing.
The demo, on real-shaped data
Review-queue records from a mid-size e-commerce shop:
queue = [
"Order 48213 held: BIN 529904, decline code 4863 on first attempt, "
"retry succeeded, shipping to address seen in 3 prior chargebacks.",
"Order 48377 held: velocity rule VEL-013, four orders in 11 minutes, "
"same device fingerprint, different cards.",
"Order 48391 released: false positive on gift purchase, billing and "
"shipping mismatch explained by gift note. Customer verified via 3DS.",
"Order 48402 held: promo FIRSTBUY20 redeemed by account created 8 "
"minutes earlier, email domain registered yesterday.",
"Order 48440 held: card declined with code 4863 twice, third card "
"succeeded. Classic card-testing tail behavior.",
]
Enter fullscreen mode Exit fullscreen mode
Dense retrieval on the identifier query:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
E = model.encode(queue, normalize_embeddings=True)
q = model.encode(["decline code 4863"], normalize_embeddings=True)
for score, rec in sorted(zip((E @ q.T).ravel(), queue), reverse=True):
print(f"[{score:.3f}] {rec[:62]}")
Enter fullscreen mode Exit fullscreen mode
[0.471] Order 48391 released: false positive on gift purchase, billing
[0.446] Order 48213 held: BIN 529904, decline code 4863 on first atte
[0.412] Order 48440 held: card declined with code 4863 twice, third c
Enter fullscreen mode Exit fullscreen mode
A record with zero occurrences of the code outranks one of the two records that contain it. The embedding is matching the general vibe of “payment trouble,” not the token. Now BM25:
from rank_bm25 import BM25Okapi
bm25 = BM25Okapi([r.lower().split() for r in queue])
for score, rec in sorted(zip(bm25.get_scores("decline code 4863".split()), queue),
reverse=True)[:3]:
print(f"[{score:.2f}] {rec[:62]}")
Enter fullscreen mode Exit fullscreen mode
[4.87] Order 48440 held: card declined with code 4863 twice, third c
[4.12] Order 48213 held: BIN 529904, decline code 4863 on first atte
[0.31] Order 48391 released: false positive on gift purchase, billing
Enter fullscreen mode Exit fullscreen mode
Exactly the two records containing the code, cleanly separated from the noise. 4863 is a rare token, and rare tokens are precisely what the IDF term in BM25 was designed to reward — thirty years before anyone embedded anything.
The point isn’t nostalgia
Nobody’s arguing for BM25-only. On the pattern queries — “gift purchase with mismatched addresses,” “card testing behavior” — dense retrieval wins just as decisively, finding cases that share no vocabulary with the query.
The argument is against the silent deletion. Teams migrate to a vector database, the keyword index quietly dies, and nobody measures what happened to identifier queries — because the recall dashboard, if it exists at all, was built from pattern-style test queries. The failure mode is invisible by construction.
Run both. Fuse ranks (RRF, k=60, no tuning). The keyword index costs kilobytes per document and roughly nothing to serve.
The expensive part of your retrieval stack isn’t the vector database. It’s the queries it silently gets wrong — and in a fraud queue, a silently missed match is a pattern that gets to repeat.
What did your recall numbers do when you turned keyword search off — did anyone even measure?
I’m Vinicius Fagundes — principal data engineer and MBA lecturer in São Paulo. I build fraud and risk analytics pipelines for e-commerce through vf-insights.com.
답글 남기기