The Scoreboard Lied. Now Sentry Shows Which Layer Broke

This is a submission for DEV’s Summer Bug Smash: Clear the Lineup powered by Sentry.

Two capture-layer bugs made a working eval harness report false failures. Fixing them exposed a third integrity seam in the scoreboard.

My first eval scoreboard looked catastrophic.

• Local llama3.2: 5 out of 6 malformed.
• Anthropic Sonnet: 6 out of 6 malformed.

If I had stopped at the summary, the conclusion would have been easy: both engines failed.

Then I opened the raw records.

Across the runs, the same word — malformed — was hiding three completely different events. One request never reached a model because the API credits were exhausted. One local response was corrupted by terminal control bytes leaking through a CLI capture path. One Sonnet response was valid JSON wrapped in markdown fences that my parser refused to strip.

The scoreboard had compressed “no model run,” “transport corruption,” and “valid answer rejected by the parser” into one red cell. Those are not three versions of the same failure. They live in different layers, require different repairs, and support different conclusions. None of them was evidence that a model had failed the task.

The scoreboard lied — not because it fabricated a number, but because it erased the cause behind it.

I had already told the first half of this story in Your Harness Will Lie to You Before Your Model Does. That title is already carrying the Smash Stories entry. This Clear the Lineup entry is the implementation sequel: the exact capture fixes, the scoring-integrity repair they exposed, and the Sentry instrumentation that now shows which layer broke before the summary gets to blame the model.

The moment the story changed

The eval had been frozen since July 1. By the time I could run both engines cleanly, I thought I was finally measuring model behavior. Instead, the first thing the experiment measured was my confidence in my own harness.

That confidence failed twice.

The local model was doing real work while the pipe mangled its answer. Sonnet was returning the structure I asked for while the parser rejected its wrapping. The important move was not a clever patch. It was refusing to defend the scoreboard once the raw evidence contradicted it.

That changed the question from “Which model failed?” to “Which layer produced this result?” The rest of the work followed from that correction.

Project Overview

memory-authority-auditor detects when one instruction in an AI agent’s memory silently overrides another. A semantic proposer (LLM) reads memory items and proposes authority changes. A deterministic confirmer checks for a verbatim citation in the source text before any proposal counts as a finding.

The eval harness runs two engines (Anthropic Sonnet and local llama3.2 via Ollama) against a frozen fixture, records every raw output and scoring decision, and writes timestamped JSON artifacts. The fixture and scoring rules were committed and frozen before the eval code ran.

Bug Fix or Performance Improvement

Two independent capture-layer bugs caused the eval harness to report working model output as malformed. Both bugs lived in one file (agents/semantic_proposer.py). Both produced the same label on the scoreboard — malformed — for completely different reasons. Neither was a model failure.

Bug 1: CLI subprocess captured terminal UI bytes as data.

The local llama path called ollama run via subprocess.run(capture_output=True). The Ollama CLI emits ANSI terminal control sequences (spinner, cursor repositioning, line-erase) that are invisible in a terminal but corrupt captured stdout. The raw artifact contained sequences like \x1b[K (erase to end of line) and \x1b[7D (cursor back 7) embedded inside otherwise valid JSON strings.

Result: 5 out of 6 cases reported as malformed. The one case that parsed cleanly produced a confirmed finding — the model was answering correctly and the pipe was garbling the output.

Bug 2: Parser rejected valid JSON wrapped in markdown fences.

The Anthropic Sonnet path returned valid JSON, but the model wrapped every response in a `json code fence. The parser called json.loads() directly on the raw text, which starts with three backticks, not a curly brace.

Result: 6 out of 6 cases reported as malformed. The raw output contained complete, well-formed proposals for every case. This is not an exotic failure — most LLMs wrap JSON responses in markdown fences by default.

Code

Single commit: 49902b2 — 54 lines changed in one file.

Full diff: git show 49902b2 -- agents/semantic_proposer.py

Change 1 — Replace CLI subprocess with HTTP API:

Before:
`python
subprocess.run(
["ollama", "run", "llama3.2", prompt],
capture_output=True,
text=True,
timeout=60,
)
`

After:
`python
request = urllib.request.Request(
OLLAMA_API_URL,
data=json.dumps({
"model": OLLAMA_MODEL,
"prompt": prompt,
"stream": False,
}).encode("utf-8"),
headers={"content-type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=120) as response:
payload = json.loads(response.read().decode("utf-8"))
`

Explicit error handling for network failures, timeouts, and invalid JSON responses replaced the opaque subprocess failure mode.

Change 2 — Strip markdown code fences before parsing:

`python
def _strip_code_fence(text: str) -> str:
t = text.strip()
if t.startswith("`“):
t = t.split(“\n”, 1)[1] if “\n” in t else t[3:]
if t.rstrip().endswith(“`"):
t = t.rstrip()[:-3]
return t.strip()
`

Applied before json.loads():
`python
parsed = json.loads(_strip_code_fence(text))
`

The frozen fixture, scoring rules, and confirmer gate were not modified. Only the capture layer changed.

My Improvements

Artifact Engine Malformed State 20260701T225629Z llama3.2 5/6 Before fix — ANSI control bytes 20260709T191930Z Sonnet 6/6 Before fix — markdown fence 20260709T190750Z llama3.2 0/6 After fix 20260709T192344Z Sonnet 0/6 After fix 20260709T202859Z Both 0/18 v1 full run, 18 cases, clean

All artifacts — including the broken pre-fix runs — are committed in path_a_eval_artifacts/ and publicly recomputable.

What changed after the fix:

• The pre-fix scoreboard said both models failed almost every case. The post-fix v1 run showed Sonnet catching 12/12 positive cases by direction and 7/12 by exact match, with 0/6 false fires. That result was invisible until the capture path was honest.
• Malformed became a diagnosable signal. Each case now records a malformed_reasons string (e.g., "http 400: Your credit balance is too low") instead of just a boolean. Different failure modes — credits, ANSI, fences — no longer collapse into one label.
• Terminal UI was removed from the data path. The transport layer is now HTTP JSON, eliminating an entire class of capture corruption.
• Reproducibility was preserved. The fix touched only capture/parsing code. The frozen fixture, scoring rules, and confirmer gate were not modified to improve results.

Integrity hole the fixed scoreboard still had:

Fixing capture made the transport honest — but it exposed a subtler problem one layer up, in the scoring itself. When a case’s output was genuinely unparseable — malformed — the harness still scored it. A malformed positive counted as a miss, and a malformed negative could count as a clean pass as long as no forbidden finding fell out of the garbage. Unparseable output was being read as evidence: evidence of a miss on one side, evidence of a clean negative on the other. It is neither.

Commit e933894 closes that. Malformed cases are now excluded from every positive, negative, catch, pass, false-fire, and ablation total. They do not disappear — each one still carries its raw output, its malformed_reasons, its counts, and its Sentry diagnostics, and the scoreboard renders it explicitly as unscored_malformed. The harness keeps the case for diagnosis and refuses to score it. Well-formed cases run through the exact same scoring path as before.

This did not move the published numbers. The clean v1 run (20260709T202859Z) had zero malformed cases across both engines, so 12/12 by direction, 7/12 exact, and 6/6 negative traps stand unchanged. This is a going-forward integrity fix, not a re-scored result — the old runner would have let a future malformed negative slip through as a clean pass, and now it cannot. Two focused regressions lock it in: a malformed positive and a malformed negative both stay recorded and both stay unscored. Focused: 2 passed. Full suite: 40 passed, 1 expected xfail.

Verify in under 60 seconds:

`bash
git clone https://github.com/keniel13-ui/memory-authority-auditor.git
git show 49902b2 — agents/semantic_proposer.py

Pre-fix artifacts:

path_a_eval_artifacts/path_a_eval_20260701T225629Z.json (search \x1b for ANSI bytes)

path_a_eval_artifacts/path_a_eval_20260709T191930Z.json (raw_output starts with `json)

`

Why observability became part of the fix

I found these failures because the harness preserved raw output and I was willing to open it. That was enough to reconstruct the incident after the fact. It was not enough to make the next incident fast to diagnose.

Before Sentry, every failure arrived at the same destination: SemanticProposerError, then malformed on the scoreboard. The summary preserved the fact that something broke while discarding the path it took to get there. I still had to replay artifacts by hand to answer basic questions:

• Did the provider reject the request before inference?
• Did the transport corrupt a real response?
• Did the parser reject valid content?
• Did the model return the wrong schema?
• Did a well-formed case reach scoring and fail there?

That is why Sentry is not decoration on top of the bug fix. The capture patches repaired two known failures. The instrumentation repairs my ability to tell the next failures apart.

I do not want another red cell that merely says malformed. I want the trace to show the provider, engine, case, raw-output preview, parser symptom, malformed reason, and final scoring decision in the order they happened. The point is not more telemetry. The point is preserving causality.

Best Use of Sentry

The two bugs described above — ANSI corruption and markdown fences — both threw the same exception class (SemanticProposerError) and both produced the same label on the scoreboard: malformed. The eval harness counted them, but it could not distinguish terminal corruption from a code fence from an HTTP 400 from expired credits. Three completely different failure modes collapsed into one boolean.

Sentry was integrated into the eval harness specifically to prevent that collapse from happening again. The integration uses Error Monitoring, Distributed Tracing, and AI Agent Tracing from the Sentry SDK.

What the integration does:

The sentry-sdk[anthropic] package provides auto-instrumentation for Anthropic API calls. For local Ollama calls, a manual span records model ID, provider, a preview of the input, and the response size — giving both engines the same tracing coverage.

Every eval run creates a root transaction (path_a_eval) with nested spans: one per engine, one per case inside each engine. Each case span records the case ID, class, engine, proposal count, confirmed finding count, and scoring result. The full eval waterfall is one trace.

When the proposer returns output that fails JSON parsing, a breadcrumb fires before the exception handler. The breadcrumb carries the raw output length, a preview of the first 200 characters, and a boolean for whether the output started with a markdown code fence — the exact diagnostic data that would have immediately identified Bug 2 without replaying artifacts manually. When a proposal is missing required fields, a separate breadcrumb fires with the list of present keys vs. expected keys.

If any case throws an exception (network timeout, invalid JSON, transport failure), sentry_sdk.capture_exception() sends the full traceback with the breadcrumb trail attached. At the engine level, if any cases were malformed, sentry_sdk.capture_message() fires a warning with the malformed count.

The audit CLI pipeline (audit_cli.py) wraps the full run in its own transaction, and the run_audit function is decorated with @sentry_sdk.trace for automatic span creation.

Why it matters for these bugs specifically:

Both bugs would have been diagnosable on the first run with Sentry active. Bug 1 (ANSI) would show up as a SemanticProposerError with a breadcrumb whose raw_preview field contained visible \x1b[K sequences — no artifact replay needed. Bug 2 (fence) would show starts_with_fence: true in the breadcrumb data on every Sonnet case, immediately separating it from other parse failures.

The malformed label collapsed three causes into one boolean. Sentry breadcrumbs carry the cause. That is the difference between knowing something broke and knowing what broke.

Integration is zero-cost when disabled. All Sentry calls are no-ops when SENTRY_DSN is not set. When the Sentry work landed (d331801), the existing 38-test suite passed without Sentry configured and no test was modified. The later malformed-scoring fix (e933894) added two focused regression tests for the exclusion behavior, so the current suite is 40 passed / 1 expected xfail.

Files changed:

File Change requirements.txt Added sentry-sdk[anthropic]>=2.0 agents/semantic_proposer.py Breadcrumbs on JSON parse failure (raw preview + fence detection) and missing proposal fields; manual AI span for Ollama audit_pipeline.py @sentry_sdk.trace on run_audit audit_cli.py Sentry init + root transaction wrapping audit pipeline path_a_eval_runner.py Sentry init + root transaction, per-engine spans, per-case spans with scoring data, breadcrumbs on malformed output, exception capture

Sentry tools used: Error Monitoring (exception capture with breadcrumb context), Distributed Tracing (transaction → engine span → case span hierarchy), AI Agent Tracing (sentry-sdk[anthropic] auto-instrumentation for Anthropic + manual spans for Ollama).

What this changed in how I trust an eval

The patches are small. The trust model changed more than the code did.

An eval result now has to survive five separate questions before I treat it as evidence:

1. Did a model actually run? A credit, authentication, timeout, or provider failure is not model behavior.
2. Did the transport preserve the response? Terminal UI output is not automatically a machine-readable data channel.
3. Did the parser reject content or meaning? Valid JSON inside common wrapping is a parser-contract issue, not a reasoning failure.
4. Did the scorer have interpretable evidence? An unparseable case cannot honestly count as a miss or a clean pass.
5. Can another person reconstruct the path? Raw output, frozen fixtures, commit history, traces, and failed artifacts have to remain available.

This is also why I kept the broken runs. The embarrassing artifact is not debris to clean up before publication. It is the receipt that proves the before-state existed. Without it, the clean run is just a better-looking chart asking to be trusted.

The malformed-score repair pushed that lesson one layer deeper. After fixing transport and parsing, I found that the scorer could still convert “we cannot interpret this output” into a judgment about the model. A malformed positive became a miss. A malformed negative could look clean. The harness was still trying to force uncertainty into a binary result.

Now it refuses. The case stays visible. The raw answer stays visible. The reason stays visible. The trace stays visible. Only the score is withheld.

That is the distinction I want this project to enforce: unknown is a real state, not an inconvenient value to round into pass or fail.

The real lineup I cleared

I started with what looked like two model failures. The actual lineup was longer:

• a provider request that never became inference;
• a terminal channel pretending to be a data API;
• a parser too brittle for common model wrapping;
• a single malformed label collapsing unrelated causes;
• a scorer willing to treat unparseable output as evidence;
• and a summary confident enough to hide all of it behind one number.

Each repair cleared one obstruction without erasing the previous receipt. The HTTP path fixed transport. Fence stripping fixed parsing. Sentry restored causal visibility. The malformed filter stopped the scoreboard from converting uncertainty into a verdict.

The models were never the only thing on trial. The provider path, capture layer, parser, scorer, renderer, and my own willingness to trust a clean table were on trial too.

That is the lesson I am carrying forward: when an AI eval gives me a dramatic result, I do not ask whether the number looks plausible. I ask whether every layer between the model and that number left enough evidence to deserve belief.

The scoreboard is the last witness in the chain. It should never be the first one I trust.

Repository: memory-authority-auditor
Bug fix commit: 49902b2 — Fix both proposer capture bugs and record first clean two-engine Path A eval
Sentry integration commit: d331801 — Add Sentry observability to audit and eval pipelines
Malformed-exclusion commit: e933894 — Exclude malformed cases from eval aggregates; malformed cases stay recorded and diagnostic, never scored