“Does your agent know what it doesn’t know?” has no answer. It has a coordinate.

Part 3 of **The Answerability Problem. Part 1 showed the standard harness excluding the questions that test refusal, and my own system scoring 0.000 on them. Part 2 measured six candidate signals and found none that beat plain cosine. This part explains why both were asking a question that has no answer. Code and every number: RE-call (MIT).

Two public benchmarks disagree about whether memory systems can abstain.

On LOCOMO’s adversarial category my system scores 0.000 out of 446, and every lever I ship reaches a usable catch rate only by refusing 60% of legitimate questions. On BEAM’s abstention category the same class of system does something, and the incumbent’s own published answers abstain correctly 38 times out of 70.

For weeks I treated that as two data points about one capability and tried to reconcile them. It is not two data points. It is two samples of a hidden variable, each reported as a scalar.

Name the variable and the disagreement dissolves.

The variable is how far the question sits from what the corpus contains

Ask a conversation log what Caroline realised, when it was Melanie who realised something and Caroline is right there discussing something else. The answer is absent by inches. Now ask the same corpus about restaurant reservations, when it contains none at all. Also absent, by a mile.

Both are “unanswerable.” Reporting one number for both is like reporting one number for “can it see” without saying how far away you put the object.

So we built a benchmark whose x-axis is that distance. Questions are generated at controlled excision distances from their own gold evidence: at r=0.00 only the single supporting turn is removed, at r=1.00 the whole surrounding topic is gone, with rungs in between. The document-id lists are frozen into the released manifest, so a system under test never computes its own distances. It just answers, and the harness already knows where each question sits.

The result is a curve, and the shape is the finding

Discrimination between answerable and unanswerable questions, by rung. AUC, n=200 per rung:

rung r=0.00 r=0.25 r=0.50 r=0.75 r=1.00 AUC 0.567 0.784 0.841 0.921 0.968

At the far rung it is nearly perfect. At the near rung it is barely above chance.

So “does your memory know when it doesn’t know?” has no answer. The honest reply is a question back: how far away? A usable gate needs roughly 0.90, and this system crosses that between r=0.50 and r=0.75, which is to say only once half to three-quarters of the topic has been deleted.

That curve also explains the benchmark disagreement. LOCOMO’s adversarials are built by misattribution, right event and wrong speaker, so they sit near r=0.00. BEAM’s are adversarially constructed too and sit near zero as well; its unanswerable questions actually score higher similarity than its answerable ones. Corpora where abstention appears to “work”, bounded technical documents where the miss is a different subject entirely, sit at the far end. Nobody was wrong. Everybody was reporting one point and calling it a capability.

The mechanism, and it is not a threshold problem

Here is the part I did not expect, and the reason I no longer think this is fixable by tuning.

Take each answerable question, delete its gold evidence, and measure how far the top retrieval score moves. Paired, same question, n=200. Mean change: −0.0111.

The mean is not the story. The distribution is:

• 130 of 200 questions: exactly zero change. Deleting the answer moved the score not at all.
• 64 decreased.
• 6 increased. Removing the evidence made the corpus look more relevant.

For roughly two thirds of these questions the top hit was never the gold evidence to begin with. The number a threshold gates on was not reading the answer. It was reading something else nearby, and it went on reading it after the answer was gone.

You cannot threshold your way out of that. A threshold is a monotone rule on a score, and the score is not a measurement of the thing you want to decide about. That is why Part 2’s six signals clustered and all failed: they were six ways of reading the same wrong quantity.

Then we checked whether any of it was an artifact

A curve measured on one embedder is a property of that embedder until shown otherwise. So the whole thing was re-run on gte-base: different model family, 768 dimensions against 384, and a cosine range that does not overlap the first one at all (0.7620–0.9332 against 0.4945–0.8238).

rung r=0.00 r=0.25 r=0.50 r=0.75 r=1.00 bge-small, 384d 0.567 0.784 0.841 0.921 0.968 gte-base, 768d 0.570 0.794 0.841 0.925 0.976

Within 0.010 at every rung, identical at r=0.50, and the 0.90 crossing falls in the same interval on both. The shape survives a change of family, size and scale. The magnitudes do not: the within-unanswerable deltas run about 2.1× smaller on gte-base. That is exactly why the predictions were registered on sign, monotonicity and ordering rather than on size. Had the pre-registration said “reproduces −0.1100”, a real effect would now be reading as a failure.

One detail I checked rather than reported: both arms have 173 of 200 questions monotone, and that identical count is coincidence. The two sets are not the same questions, overlapping on 156. What is substantive is that 90% of the questions monotone under one embedder are monotone under the other, which is agreement per question rather than in aggregate.

Two admissions, because they are the useful part

The first version of this benchmark passed, and the pass was garbage. v1 reported a perfect +1.000 discrimination across the board, which I came close to publishing. At the largest excision the removal took out the entire conversation, and the slice under test was that conversation, so the index was empty. It scored a perfect ability to detect that nothing was there, on nothing being there. Every non-empty contrast in that arm was flat, at most +0.003.

And I predicted the wrong shape. My registered prediction was a step: some distance at which the system flips from confident to uncertain. It is a gradient, smooth and monotone, and the shipped binary abstention decision is blind to every bit of it.

A third admission belongs to the pre-registration rather than to me, and it is the honest reading of the whole arm. Before running, it recorded: if the gradient holds, the most likely reading is mundane, since any dense retriever’s top-1 similarity falls as you delete the relevant documents. That is what happened. The mechanism is almost certainly generic to dense retrieval rather than a property of my system. The interesting content is not that the gradient exists. It is that a graded signal sits underneath a binary decision that cannot express it, and that at the boundary itself the signal is worth almost nothing.

The guard that cannot fire

While measuring the above, the shipped abstention floor turned out to be inert.

The default constant is a cosine of 0.50. On bge-small the observed scores span [0.4945, 0.8238], so 1 response in 1200 falls below it. On gte-base the entire distribution sits above 0.76, and the answer is 0 of 1200.

The guard my library ships, whose job is to withhold an answer, could not fire on either embedder. Not “fired rarely”. Could not fire, because the constant sits below the whole distribution it is compared against.

That is the fifth instance of this pattern I have found in a month, and the first one located in the shipped library rather than in a benchmark harness. It is embedder-specific in both directions: on a paid API embedder the same constant does the opposite, starving 23.3% of legitimate queries. One number, three behaviours, depending on a model choice it knows nothing about.

What this does not establish

• Not a quality comparison. Nothing here says one embedder retrieves better than the other. Only the shape of the distance response was measured.
• Two families is not “embedder-independent”. It raises confidence, and says nothing about API embedders, where the same constant fails in the opposite direction.
• Not a single-variable change. Family, size and dimensionality moved together, so the agreement cannot be attributed to any one of them.
• Same corpus, no judge. A shared-corpus artifact would reproduce here rather than be caught, and answer correctness is unmeasured throughout.

What I would like argued with

The claim I am least sure of is the strong one: that abstention is not a property a memory system has, but a function of distance that has to be measured against an axis, and that any single-number abstention score is therefore a category error, including the ones I published earlier in this series.

If that is right, “does it know what it doesn’t know?” should stop being asked, and every benchmark reporting one abstention number should report a curve instead.

If it is wrong, the cheapest way to show me is a signal that separates at r=0.00, where mine sits at 0.567 on two independent embedder families. The manifest is frozen and public, the harness is one command, and it costs nothing to run.

Part 4 is the ledger: what this architecture buys, what it costs, and where it loses on purpose.

The ladder, its pre-registrations and every artifact: RE-call, benchmarks/ladder/ and results/ladder/. Both verdicts, including the retracted v1, are committed rather than summarised.