Killed by the Word 'git': One Token of Coincidence, 40 Points of Pass Rate

Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

The article that split extraction from replay ended on a worked example: F-001, the git case. The model extracted the right rule — “when git push fails with non-fast-forward, pull latest changes before pushing” — almost verbatim. Replay scored it prevented 5, broken 3, and the verdict was INCONCLUSIVE. The three “broken” successes were S-022-git-pull, S-023-git-status, and NM-044-git-commit-hook. A successful git status counted as broken by a git-push rule because the two share the token git.

That was the diagnosis. This is the fix, and the 40+ points it bought.

The bug: “broken” counted coincidence as interference

The scorer’s job is to decide whether a rule does net good: does it prevent more failures than it breaks successes? The broken count is supposed to mean “this rule would have fired when things were working” — genuine interference. But “would have fired” was computed with the same lexical matcher as everything else. Sharing a token is a match, so:

every success that merely rhymes with the rule counted as a success the rule broke.

For a domain-heavy corpus like failures/positive, where git rules sit next to git successes, that is not a rare edge case. It is the dominant failure mode. Combined with a precision < 0.5 bar and a zero-tolerance near-miss penalty, the result was failures/positive at 8–10% pass for a release: correct rules, systematically demoted by their own neighbors.

The fix (#723 + #724): a margin, an ordering, and a definition

Domain-gate broken. A success can only break a rule if it is in the rule’s own domain. Cross-domain coincidences stop counting outright.

Require a match-strength margin. A same-domain success counts as broken only if it clears threshold + 0.10. The one-sentence version of the principle:

a near-threshold success match is coincidence, not interference.

A git status success that only weakly matches a push-specific trigger is no longer a “break.” It is two git commands sharing vocabulary, which is what they are.

Reorder the scorer and bound the near-miss band (#724). The old ordering let the zero-tolerance near-miss penalty over-fire on rules that prevented more failures than they broke. The new order: no-signal → inconclusive; broken > prevented → fail; near_misses > 2 → inconclusive; otherwise → pass. The definitional shift: “breaks fewer successes than it prevents failures” is the definition of a useful rule, and the scorer now admits net-positive rules (precision ≥ 0.5) instead of downgrading them.

Two supporting fixes made the candidate pool honest while we were in the neighborhood: recall-weighted ranking instead of precision-first (#731), and signature-aware 2-pass dedup — roughly 40% of the two-pass candidates were identical, so the runner’s “best” candidate disagreed with what production would have picked (#732).

The data

failures/positive (n=50), both cloud models:

v0.3.0 v0.3.1 gpt-4o-mini v0.3.1 llama-3.1-8b Pass rate 8% / 10% 50% [0.37, 0.63] 52% [0.39, 0.65] Precision — 0.544 0.554 Recall — 0.221 0.240

Gpt verdict breakdown on the corpus: pass 25, no_signal 17, blocked_by_broken 3, blocked_by_near_miss 4, min_sample 1. The residual ~44% inconclusive is mostly no_signal, and the report attributes it to the same reference-pool precision limit we explicitly accepted on raw/synthetic (J12): correct triggers each breaking 1–4 generic same-domain successes. Widening the margin further to rescue those would loosen a real safety trade-off, so we documented the limit instead of tuning past it.

Whether F-001‘s specific candidate now passes lives in the committed per-trajectory artifacts; the corpus-level number above is the claim I am making, and it is the one with a confidence interval.

The proof the other side of the gate still held

Loosening a positive gate is only a win if the negative gate holds, and v0.3.1 made that a measured claim instead of an aspiration:

Safety measure v0.3.1 result nearmiss (rejection corpus) 27/50 gate-silenced by recovery detection; of 23 active, 0 accepted — Wilson 95% CI [0.857, 1.000] adversarial/* (9 corpora) 0 accepted across injection, misleading, contradiction, unsafe, poisoning, tool_output_injection, compounding_multiturn, unsafe_realistic, and both harmbench slices — both models Generic triggers 0.4% (gpt) / 0.7% (llama) vs <10% target Total pass volume 116/119 → 601/626 — ~5×, with safety flat

Three structural pieces make that table mean something:

1. J1 — rejection corpora are scored as rejections. nearmiss and adversarial/* pass iff accepted == 0. The zero is a first-class measured quantity with a CI, not the absence of a logged failure. (v0.3.0 scored these through the extraction branch — an inverted gate that made 1 false accept read as a pass. The fix is one commit; the lesson is that “pass” must be defined per corpus type.)
2. The source-trust gate left the test harness. v0.3.0’s #727 documented an adversarial trust gap as test-only. #775/#776 wired the source-trust check into production auto-promotion — a rule whose evidence traces to an untrusted source is rejected before it can be promoted, in the product, not just in the field test.
3. The shipped container agrees. The Docker field test went 159 → 180/180 (+21 new regression tests covering the 43 M2 code-review fixes, #762–#804), so the scorer, gate, and trust checks hold in the deployment artifact, not just the dev tree.

What worked

• The margin principle. “Coincidence, not interference” is one sentence, one constant (0.10), and worth most of the 40 points. Cheap fixes that encode a real distinction beat expensive fixes that encode a threshold.
• Defining “useful” before scoring. Net-positive — prevents more than it breaks — became the promotion standard the scorer implements. When the definition and the scorer agree, the scorer can be simple.
• Zeros with CIs. 0/23 accepted with a Wilson lower bound of 0.857 is a claim. 0/23 accepted with no denominator is a hope.
• Loosening one side while measuring the other. The pass-rate jump and the safety table are the same release. Reporting both is what makes either of them credible.

What didn’t work

• Token overlap as interference. The original broken count treated lexical match as causal. Two texts sharing a word is a fact about the texts; “this rule broke this success” is a claim about events. The scorer was grading the first and reporting the second.
• Zero-tolerance near-miss penalty. It over-fired on exactly the rules worth having — the ones that prevent more than they break. Bounding the band at > 2 restored the distinction.
• Precision-first ranking. It favored low-recall rules and starved the corpus of the specific, high-recall ones. Ranking should optimize what you promote.

Questions we still can’t answer

• Is 0.10 the right margin, or should it be per-domain? Git domains are token-dense; browser-tool domains may not be. We have one margin and no curve.
• Can “0 accepted” be claimed where the model goes silent? J16: llama-3.1-8b returned zero candidates on unsafe_realistic 13/20, contradiction_harmbench 10/15, and misleading_harmbench 4/15. Those zeros are lower bounds — silence is not rejection. The gate is safe either way, but the report must not dress a non-answer as a measured rejection.
• The two protocol metrics we still haven’t run. Cross-session repeat-failure reduction (#741) and human-vs-replay agreement (#742): tooling complete, protocols not run. They are the difference between “core gate closed” and “full gate closed,” and neither is a blocker for the core promotion path.
• Where does J12 stop? The accepted raw/synthetic precision limit is real, but “accepted” is a boundary we drew. Re-drawing it is a safety trade-off decision, not an engineering one — and it hasn’t been made by anyone with more information than us.

What I learned

Overlap is not interference. A relation between two pieces of text is not a relation between two events. If your scorer counts “shares a token” as “broke,” add a margin or a domain gate — demand more than coincidence before counting a collision.

Loosening a gate is testable, and the test is the negative corpus. The 40 points we gained on failures/positive are only news if nearmiss and the nine adversarial corpora say zero afterwards. Run them in the same sweep, and publish both tables in the same report.

“Net positive” is a definition worth stealing. A rule that breaks fewer successes than it prevents failures is useful, full stop. Most scorers I have seen implement some muddle of precision and recall instead of that sentence. The sentence is simpler and it matches the intent.

Report zeros with their denominators and their CIs. 0/23, [0.857, 1.000]. The lower bound is the honest number, and it is the number a skeptical reader will recompute.

The broader lesson

When a gate kills your good outputs, find the single relation it is mis-grading — here, token overlap masquerading as interference — and replace it with a relation that costs almost nothing to compute: same domain, plus a margin. Then prove the other side of the gate with a measured zero, not a quiet one.

That closes the v0.3.1 arc: an honest ruler, a full haystack, a reachable floor, and a judge that finally distinguishes coincidence from interference. What’s left is the work the report names but the release does not: the 0-accepted corpora, the cross-session protocol, and the human-agreement run — the difference between a gate we closed and a gate we could close.

References

• CauterRule v0.3.1 release notes
• v0.3.1 field test report (Fix 2 + Fix 3, §7 safety metrics, Appendix A J1/J12/J16)
• Docker field test results (180/180, +21 regression tests)
• SECURITY.md (source-trust gate, #727)
• v0.3.0 field test report (the F-001 diagnosis)
• User guide · Changelog

CauterRule v0.3.1 is released. The scorer before/after, the safety CIs, and the J1/J12/J16 issue journal are in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes