When I shipped v0.2.1 of PlannerCritic, I thought the hard part was over. The engine had survived a 170-goal field test across 40 domains. The published results looked strong: 73 of 73 balanced goals approved, 96 of 97 strict goals escalated, 8 of 8 adversarial goals blocked, 30 verdict deltas all attributable, and zero new engine issues discovered by the sweep.
That was the release story I believed.
Then a stranger on the internet checked it in public and found that the engine was mostly right, but parts of the story I wrote about it were wrong.
That distinction matters much more than most maintainers want to admit.
The Offer That Changed the Release
After I published the field-test article, a reader offered something unusual. He did not just leave feedback or point out one suspicious sentence. He proposed a protocol: freeze a small set of claims from the release before inspection, verify them only from public artifacts, and record any divergence as a divergence instead of explaining it away.
That is a much stricter process than most OSS releases get. In practice, many of us release from a mix of memory, local state, CI output, and confidence. We know the code. We know roughly what passed. We know what we meant. What we usually do not do is force our public claims to survive independent reconstruction from the outside.
So I said yes.
We froze five claims from the v0.2.1 release:
- the repaired adapter-import test really exercised imports,
-
approving_authoritywas test-proven but not reachable from shipped surfaces, - 1295 deterministic tests passed,
- the field test had zero true failures,
- the 30 verdict deltas were all attributable.
The important detail here is not the number five. The important detail is that the claims were frozen before inspection. That removed the easiest escape hatch: adjusting the claim after seeing the evidence.
What the Audit Found
Two of the five claims came back clean. Three did not.
The first failure was the easiest to explain and the most annoying to defend. I said 1295 deterministic tests passed. The public CI run on the release commit showed 1294 passed, 1 failed, and 14 skipped. The failed test was a flaky SQLite concurrency case. Operationally, that meant the release was still understandable. But documentation-wise, it meant my claim was cleaner than the public evidence supported. That discrepancy became #263.
The second failure was worse because it was not a flaky detail. It was a contradiction in the release document itself. In one place, my v0.2.1 field-test report said “zero true failures.” Two tables down, Scorecard B recorded True Fail = 1. I had written both statements. I had published both statements. I had not noticed that they were incompatible. That became #246.
The third failure was smaller but revealing. The release documentation said plan_oscillation_detected fired for 3 goals. The actual stored artifacts showed 5. That became #247.
None of those findings meant the engine was broken. What they meant is more uncomfortable: the engine can be right while the release around it is wrong.
The Engine Was Stronger Than the Release Notes
This is the lesson I took most seriously.
The underlying PlannerCritic results from v0.2.1 were still solid. The field test really did find zero new engine issues. The 30 verdict deltas really were attributable. The inherited corpus behavior really did hold. The live boundary evaluator really did return label_flip_rate=1.0, evidence_drift_rate=1.0, family_migration_rate=0.0, and underclaim_approvals=0.
The problem was not that the system was wrong. The problem was that the release package around the system had drifted out of sync with the evidence. Prose and scorecards diverged. Counts were copied across documents without a final reconciliation pass. My local confidence ran ahead of the public artifact chain.
That is not a glamorous bug class, but it is exactly the kind of bug that undermines trust in serious infrastructure projects. A release note is part of the product surface. If it overstates what the evidence says, even slightly, readers stop trusting the stronger claims too.
What Changed in v0.2.2
The public audit did not just embarrass me. It materially shaped the next release.
v0.2.2 started by fixing the exact class of mistakes the audit exposed. #246 fixed the “zero true failures” contradiction. #247 reconciled the oscillation count. #248 removed 75 committed *.py,cover artifacts from src/. #263 reconciled the 1294/1295 test-count discrepancy. #264 added a failure-origin taxonomy so defects could be tied to the layer that should have caught them first.
That work became the M1 foundation milestone of v0.2.2.
By the time v0.2.2 closed, the evidence chain was much tighter. The release finished with 183 of 183 goals complete across 43 domains, 1347 tests passed, 15 skipped, 91% coverage, and the inherited top-level contract still intact: 73/73 balanced approved, 96/97 strict escalated, 8/8 inherited adversarial blocked. The release also added 13 new security fixtures, found a real boundary regression on the first run, fixed it, and reran the benchmark until the published result was clean.
The engine improved, but the bigger improvement was procedural: the release got harder to fool.
The Real Value Was the Protocol
What stayed with me was not just that a reader found discrepancies. It was that the verification process itself was better than the one I had implicitly been using.
Freezing claims before inspection matters because it prevents quiet drift in the story. Restricting the check to public evidence matters because it forces reproducibility. Recording partial confirmations honestly matters because it keeps the release from becoming a negotiation between intention and evidence.
That last part is especially important for AI-adjacent systems. It is easy to produce polished prose around LLM systems. It is easy to produce screenshots, dashboards, and pretty summaries. It is much harder to keep every sentence tied to a durable artifact that another person can reconstruct without talking to you.
The maintainer is usually the worst person to verify the release narrative they just wrote. They know too much. They know what the system was supposed to do. They know what the local run looked like. They know which discrepancy feels “basically fine.” All of that context makes them less reliable as the final authority on the public record, not more.
What I Would Recommend Now
If you maintain an OSS system with a strong performance or safety claim, I would suggest a simple rule: pick three to five release claims and force them to survive a hostile public replay.
Not your own replay.
Someone else’s.
And make those claims specific enough to fail.
If the claim is “the release is much better,” nothing useful happens. If the claim is “1295 deterministic tests passed on the release commit,” the claim can be checked. If the claim is “30 verdict deltas were all attributable,” the claim can be checked. If the claim is “the report contains zero contradictions,” the claim can be checked.
That is the kind of pressure that improves releases.
The Question I Care About Now
Most OSS maintainers ask how to get more feedback. I think the more useful question is this:
Which claims in your release would still survive if a stranger froze them and verified them from public evidence only?
That is a harsher standard than green CI.
It is also a better one.
Previous PlannerCritic articles
- Article 1: I Ran 157 Agent Plans Against a Real LLM
- Article 2: I Told My LLM Critic to Be Adversarial
- Article 3: The Planner Made the Same 3 Mistakes Every Time
- Article 4: I Ran 170 Agent Goals for $0.49
- Article 5: I Tried to Prompt-Inject My Own Agent Engine
Links
- Repo: https://github.com/deghosal-2026/planner-critic-engine
- v0.2.1 field-test report: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/field-test/v0.2.1/field-test-results-0.2.1.md
- v0.2.2 field-test report: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/field-test/v0.2.2/field-test-results-0.2.2.md
- v0.2.2 release notes: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/reference/release-notes-v0.2.2.md
- Issue trail: #246, #247, #248, #253, #263, #264
Next in the sequence: My Agent Said No 96 Times. The Most Valuable Output Wasn’t the Plan.