두 개의 동일한 실행은 89와 89를 기록했습니다. 두 건이 뒤집혔다.

Read on: the A/B test this corrects · 繁體中文版

Two weeks ago I published an A/B test: I cut 41 AI tools’ self-descriptions roughly in half, then ran a behavioural question bank against both versions to check that trigger rate hadn’t dropped. Before: 88/96. After: 90/96.

I wrote this sentence about it:

±2 cases at this sample size is noise, so I am not claiming it got better.

A reader named Vinh Nguyen replied, and the reply amounted to one question: how do you know?

He was precise about it. Three votes per case, majority wins, so any case sitting near the 2-1 boundary can flip on sampling alone. There was a between-arm delta with no within-arm noise floor to measure it against. And “three up, one down” — exactly what my four movers looked like — is the shape churn produces.

He was right. That sentence was an assertion, not a measurement. It is also the same move the post spends two thousand words criticising: claiming a property of a system without observing it. I made it in the one paragraph where I was grading my own work.

So I ran his experiment. The part I didn’t expect is that it cuts in two directions at once.

First I went looking for the answer I’d already paid for

Before spending anything, I checked whether I could bound the churn retroactively from the run I had already done. The harness makes three calls per case and takes the majority, so it necessarily computes “2 of 3” or “3 of 3” somewhere. Count the 2-of-3 cases and you have an upper bound on how many could flip — for free.

I couldn’t. And the reason has a nastier shape than “it wasn’t recorded.”

The harness does compute the vote split, and it does print it — for cases that failed. A case that passed 2-1 prints nothing at all. The JSON output aggregates to per-skill pass/total and drops every split.

So the one shape that actually flips under sampling — the narrow pass — is precisely the shape no layer of that tool had ever shown me. The instrument was blind in the exact direction that produces reassurance. Failures got scrutiny; lucky wins got a checkmark.

I fixed that first, because it is the cheap half and it makes every future round free to interpret. Per-case vote counts now survive into the JSON, and the human-readable run ends with a count of non-unanimous cases. Then I checked the fix the way I should have checked the original claim: forced every case to 2-of-3 and confirmed the new line moved from “0 split” to “4 split.” It reads the real votes. It isn’t printing a constant.

The baseline

Same commit, same question bank, same model, same machine, same afternoon. The unchanged arm, twice, with nothing touched in between. 96 cases, 288 calls per run.

Run 1 Run 2 Total passed 89 / 96 89 / 96 Cases that flipped between the two runs 2

Read those two rows together, because the gap between them is the whole point. The score is identical. Two cases flipped anyway — one up, one down, cancelling exactly.

If I had run the same arm twice and looked only at the total, I would have concluded the harness was perfectly stable. It isn’t. It has a floor of about 2, and the total-score view is structurally incapable of showing it, because flips in opposite directions erase each other before they reach the number I was reading.

So my +2 sits exactly on the floor. The sentence in my post happened to land on the right answer for a reason I had not earned.

Where the noise actually lives

This is the part I’d want if I were reading someone else’s post.

My question bank has three kinds of case per skill: should-trigger, should-NOT-trigger, and a boundary case. Splitting the two flips by kind:

Case kind Count Flipped should-trigger 48 0 should-NOT-trigger 24 2 boundary 24 0

All of the variance is in the negatives. Zero flips in 48 positive cases, zero in 24 boundary cases, both flips in the 24 negatives.

In hindsight it’s obvious. “This sentence is obviously asking for that tool” is a strongly determined decision. “This sentence should not reach for that tool” is a decision made against 116 competitors, and it’s soft.

Which means my question bank spends 75% of its budget on the categories carrying none of the noise. If I want a future A/B to be readable, the fix is more negative cases, not more rounds. I would not have guessed that, and no amount of staring at the total would have suggested it.

The bad news, which is also the interesting news

I flagged one skill in the original post as “the one to watch” — the one I had compressed hardest, which lost a case.

It’s still down. Both of today’s runs put it at 3/4, and the case it loses comes back 0 of 3 — unanimous failure, twice, seventeen days later. And it is a positive case, which is the category where the noise produced zero flips all day.

To be sure that signature meant what I thought, I ran a deliberate control. I took six skills whose descriptions the original disciplined trim could only compress by under 20% — meaning they had almost no redundancy left — and force-cut each by roughly 64%, the same ratio as the one that broke. If over-trimming has a signature, this should produce it.

It did, loudly. Those six went from 23/24 and 22/24 down to 19/24, and one of them collapsed from 4/4 to 1/4. Three separate cases fell from 3/3 to 0/3 or 1/3, every one of them positive or boundary — none in the negatives where the noise lives.

So real damage and churn do not look alike once you can see the votes:

• Churn is a negative case wobbling between 1-of-3 and 2-of-3.
• Damage is a positive case collapsing to unanimous failure and staying there.

A total score cannot tell those apart. That’s not a subtle limitation. In my case the aggregate said “no systematic decline,” and it was true, and it was concealing a genuine regression, because one real loss and several noise-driven gains summed to approximately nothing.

The threshold question

The other commenter, Kane Lim, proposed that the 64% case suggested a practical threshold — a point where compression starts removing semantic signal rather than redundant text.

The control answers this, and sharpens it. The same 64% cut cost one case on the skill in the original post, and three cases on a skill in the control. The ratio was identical. The outcome was not.

The difference is what the 64% removed. The skill I could compress that far with discipline could be compressed that far because most of its 689 characters were restatement of a six-layer rule. On a description that is already tight, reaching 64% means cutting into signal — there is nothing else left to take.

Which inverts how I had been reading the number. Compression ratio isn’t a risk driver. It’s a measurement of how much redundancy was there to begin with — and I had been treating a symptom as a cause. The two variables aren’t merely confounded, they’re mechanically linked: how far a disciplined trim can go is the redundancy. Across my 24 skills the achievable ratio ran from 3% to 64%, and exactly one exceeded 49%.

And a correction I owe the original post. I wrote: “everything I cut by 30% held steady. The only one that moved was the one I cut by 64%.” That’s wrong on my own data. The four skills that moved were cut by 64%, 49%, 48%, and 17%. The 17% one moved by exactly as much as the 64% one. The table proving it sat two paragraphs above the sentence.

What I’d take from this

1. Run the same arm twice before you compare two arms. It cost about a dollar. It reclassified two of my conclusions.
2. A total score is a lossy summary, and it’s lossy in a specific direction: opposite-direction flips cancel, so the aggregate systematically understates instability. Two runs scoring 89 and 89 hid two real flips.
3. Record the vote split per case, not just pass/fail. Anything that takes a majority is discarding the number that says how close it was. And if your tooling prints the split only for failures — mine did — then the flip-prone narrow pass is the one thing you can never see.
4. Sort your cases by kind and check where the variance actually sits. Mine was 100% in one category holding 25% of the budget.
5. A stable unanimous failure is readable at n=1. You don’t always need a bigger sample; sometimes you need a less lossy instrument. 0-of-3 twice is a signal. 2-of-3 once is not.

What this doesn’t establish

Two runs give a point estimate of the floor, not a distribution — I know the floor is about 2, not that it is 2. The over-cut control was a single arm, single round; its individual collapses are strong, but its total is subject to the same ±2. The six skills I over-cut were selected by a proxy — “the disciplined trim could only compress them under 20%” — which measures how tight a description is, not how narrow a skill’s scope is, and scope was Kane’s actual question. I also chose how to cut them, keeping the opening trigger sentence and dropping identifiers and symptom phrases; a different cut might land differently.

One last thing, since it’s the same failure as everything above. When I went to report what these runs cost, I found the harness doesn’t print its cost block in JSON mode — that code sits in the other branch. So the actual spend for the three runs was never recorded, and the roughly US$1.2 I quoted is extrapolated from a single-skill run. Third instance in one day of the same defect: a number that was computed, and then dropped by one output path.

That’s the thing worth keeping. Not “measure your noise floor” — that’s just good practice. It’s that my instrument computed the answer to Vinh’s question, printed it selectively in the direction that flattered me, and dropped it everywhere else. Nobody was going to find that from the outside. It took a stranger asking one question I couldn’t answer.

本文原載於我的部落格：Two identical runs scored 89 and 89. Two cases had flipped.