잘못된 것을 디버깅하려면 어떻게 해야 하나요?

Yesterday afternoon a runtime I wrote created 70 paragraphs on the same draft article. It was supposed to create one. The runtime was an experiment in explicit state, the kind where the model gets the task spec, a small JSON, the last observation, and nothing else; it had a commit rule I was proud of: nothing counts as done until a read-back confirms it. After every paragraph it read the draft back, found no paragraph, and asked the model for another one. Seventy times.

My first suspect was the model. It was the last thing that had produced output, and blaming it costs nothing. Then I looked at the response sizes of the read-backs: 742 bytes, 742, 742, twenty-five times in a row, then 131,991 bytes at once. The paragraph list had been served from a cache with a 60-second lifetime. The model had done exactly what a correct runtime told it to do, and the runtime had been lied to by its own read path. Not the model. Not even the rule. The order of two reads.

One line of context so you know where I stand: I run engineering at GoodBarber, an app platform, and I operate a production MCP server plus a few scheduled agents of my own. In the write-safety piece I asked whether a 200 meant the right thing happened; in the determinism piece I argued that verification is the job now. This one is about what comes after the verification says no. A system that is allowed to be wrong cannot be debugged by asking whether it was wrong. It has to be debugged by three questions, in this order: what was it allowed to get wrong, which layer got it wrong, and how long did the error live.

Write down what it is allowed to get wrong

I had never written this down. Doing it took an evening and changed what I look at first.

block output tolerated never allowed detected today by when it does not run MCP server, as its operator tool results with a policy envelope, one usage row per call latency; a read served from the 60-second cache when no write preceded it; a client choosing the wrong tool; 429 above 1,000 calls a minute a 2xx for a write that did not persist; a stale read right after a write by the same token; a tool silently missing from a client’s list; a push nobody scheduled Sentry on unhandled exceptions; status and error code in the usage row; one guard test against schema shapes that hide tools loud: 421, 503, 429. Silent: the server is up and its cache is wrong content agents, Claude Code sessions writing to seven blogs drafts in nonpret, patches, JSON-LD a clumsy sentence; a meta title to redo a live article patched without a diff; a create that defaults to published; an em-dash in a slug; invalid JSON-LD on a live page; an empty body reported as done my approval in the CMS; a diff before and after when a patch touches live pages; JSON-LD validation after injection nothing ships. The risk is the half-written object scheduled tasks: noon ops review, two code reviews, glossary sentinel a DM to one colleague; two morning reports; a Monday report on 170 pages a false positive dismissed in a minute; a missed run if the next one says so “all green” over a broken page; a number in the DM that is not in Nagios; a skipped run with no trace 22 negative test cases and a positive control on the sentinel; the colleague reading the DM; nothing counts the runs silence, indistinguishable from “nothing to report”

Two admissions fell out of the table. First, I have no fixed review routine: I read what an agent produced when it produced it, or when something surfaces, with no schedule and no written verdicts. Second, half of the “never allowed” cells had no detector the day I wrote them. Write it down, or every output is a bug and a feature at the same time.

Attribute before you debug

Here are the incidents I could date on my own systems since April, coded by the layer that was actually wrong. The list is what I could date, not a sample, and the split describes this list, not my fleet.

when what layer first suspect April 10 to 14 access tokens set to 300 seconds “just in case”; every client logged out every five minutes configuration the clients April 8 to August 5 a per-app session cap counted sessions for an hour after their last call; clients that never sent the protocol DELETE starved it and legitimate traffic got 429s harness the clients, again June agents patched translations on seven live blogs with no draft step; a diff afterwards found 46 damaged paragraphs (from my notes; the artefacts are gone) harness the model June a translation pipeline reported green on a French draft with a title and no body; I remember the run as green, its history is purged, the draft is still empty harness the model June 5 to 22 a schema change made for a directory review put a oneOf at the root of one tool; clients that drop such tools stopped seeing it for 17 days tool contract the clients July 29 and 31 an unbounded Redis pool hit 10,000 clients, 3,650 connections rejected, every worker locked out; two days later one stream read per session filled the 200-slot pool infrastructure the load August 5 a delete succeeded and the immediate read-back returned the object; fixed on August 10 by keying the read-after-write bypass on the token instead of a per-session variable cache the model, for a minute August the first version of my llms.txt annex read 8 days of logs and reported them as 121 the debugger the logs September 3 a state runtime created 70 paragraphs because its second read went through the cache cache, then the runtime’s read order the model June 3 to September 2 125 calls to 34 tools that do not exist, including GBContent.getItems(sectionId, opts, onOk, onErr), from 5 apps the model, with a harness that let it retry nobody, until I counted

Ten lines. The model is the direct author of one of them, the invented tool names, and even there a client let it retry the same name until it gave up. My first suspect was the model in four of the ten. The reflex was wrong four times; the model, once.

The rule that fell out of it: attribute before you touch the prompt. Data and credentials first (was the input what you think it was), then tool and cache (did the read-back read the world or a copy of it), then harness and scheduler (did the loop do what the loop does, retry, truncate, reorder), then the model, then the human reading the output. Every layer above the model is deterministic and checkable in minutes. The prompt is where you go last, because it is the only layer where a fix cannot be verified.

The transcript is a witness, not evidence

When an agent has been wrong, the transcript is the first thing you open and the last thing you should trust. It records what the model saw and what it said. It does not record what happened. Ask the transcript “what did you call, with which arguments”, never “did it work”.

On September 3, 2026, I measured what our own read path answers, 50 cycles on a test app, every article created as a draft and deleted after. Create, read, delete, read: the immediate read after the delete still returned the object 3 times out of 50, after the August fix. Once in 50, the immediate read after a create did not find the new article at all. Then the sequence an agent actually produces, create, read, read, delete, list, read: the deleted object came back 50 times out of 50, and stayed for a median 61.1 seconds, maximum 61.2. The server bypasses the cache for exactly one read, the one that follows the write; a list in between resets it. A transcript of that agent would show a delete with a 200 and a read that returned the article. Both true. Neither what happened.

The same cycles confirmed three things I had logged in August as tickets. A delete returns only the policy envelope, no deleted, no id, no status; the transcript literally cannot show what was deleted. cms_create_article still defaults to published, so a create without an explicit status is a publication. And the verification hints on a create list cms_get_article, cms_get_article_paragraph and cms_get_event, which are the first three read tools of the family in alphabetical order, while the tool plan points its verification call at cms_list_cms_sections. Follow the plan to the letter and you verify an article by listing sections.

None of this is exotic, and that is the point. The June agents that damaged 46 paragraphs were not lying. They were reporting. Nobody had asked the state, only the transcript.

Measure how long the error lived

Error rates are the number everyone asks for and the least useful one I have. What changes behaviour is time to detection, per incident, per channel.

error lived caught by stale read after delete, list in between 61 seconds, every time a script, September 3 the annex that read 8 days as 121 hours me, because a number looked too round the 17-day invisible tool 17 days a guard test written on June 22; who noticed first is not in the log 429s on legitimate traffic from the session cap until August 5; the cap dates from April 9, the first victim is undated logs, August 5 the empty French draft since June, still there nothing; I keep it ops review not running on August 8, 9, 15, 16 26 days for the first one me, counting transcripts on September 3 125 calls to tools that do not exist 92 days a CSV export, September 3

Two rows have no detector at all, only a person who happened to count. The four missing ops reviews were two closed-laptop weekends; nothing said so, because a run that does not start writes nothing, and nothing is the same colour as green. The invented tool names errored 125 times in front of a server that logs every call, and the log was never read for that question. Time to detection is the metric that tells you which of your checks are actually running. We do not lower the error rate. We shorten how long an error lives.

What survives at three weeks is the other half of that measure. For my scheduled runs: the full transcript, locally, with token counts and tool calls, so I can answer “what did it do, in what order” for any day since August. For the server: one row per call with keys and sizes, never values, and no fingerprint of which agent made it. For the June pipeline: nothing; the execution history is purged, and my memory of a green run is the only witness. Three systems, three answers to the same forensic question, and none of them can answer “was it the right action”. Only the read-back can.

The debugger is also non-deterministic

The uncomfortable part of debugging a system that is allowed to be wrong is that the thing you debug it with is allowed to be wrong too.

My llms.txt annex read the wrong window. My first E2 verifier divided every French body by an English length of one and failed six correct drafts. And the glossary sentinel, before it was a script, was a cloud routine created on August 12 that rewrote its own checker at every run: a verifier that is a slightly different program each week cannot be debugged, because there is nothing stable to debug. It ran twice, both green. I replaced it on August 17, on that design argument alone, with a frozen script, 22 negative test cases, a positive control, under version control. Four reports since, all green on 170 pages, 89 to 166 seconds each. The state is the sitemap. The validator is the test file.

Freeze the verifier first. It is the only component whose correctness you can actually prove, and the only one nobody wants to write. On one article, 20 identical translation calls at temperature 0 on September 3 gave one distinct output; the eight deterministic checks I run on a translation caught all six faults I injected, and one of them, an altered URL, was caught by exactly one check. A judge would have shrugged. The status code would have said 200.

What is not built

Not shipped on our server, still: a heartbeat on scheduled runs; a token fingerprint in the usage row, so that “which agent” has an answer; an outcome flag next to the status code, so that a 200 that did the wrong thing shows on a dashboard; idempotency keys, version-bound writes, a plan-then-commit object; the three tickets above, open since August 5, unchanged at our August 28 commit. What is built: the error budget table, the incident registry, the frozen sentinel, and a read path I now know to measure rather than trust.

When your agent was wrong last time, what did you look at first, the prompt or the data? Genuinely curious which way the reflex goes, and how often the model turned out not to be the culprit. I will answer with the registry and the read-back numbers, cycle by cycle.