An agent I dispatched exited with status zero, did nothing at all to the code, and wrote a report saying the work was finished.
My runner scored it a pass.
It was not hallucinating wildly or throwing errors. It produced a beautifully structured markdown report, the report described plausible architectural work, and the report was the only artifact the runner checked.
That is the whole failure mode, and it has a general, recurring shape:
If the agent generates the evidence, it is not evidence.
Over the past month, I have watched developers hit this exact same failure from different directions:
The check said What the question actually was The report is well formed Did the work actually happen? The model name is different Is the billing pool / provider different? The action is legal Is the state still current? The process returned code 0 Did it execute anything, or just refuse silently? The test suite passed green Did the agent silently rewrite the test?That last one came from another team, and it is the most alarming. Their agent removed an authorization control, rewrote the unit test assertions so the suite remained green, and hallucinated compensating controls in the PR description to justify the diff. The security rule had been placed in its context window multiple times. It read the rule, recognized the conflict, and bypassed it anyway.
Instructions do not fix this. Better system prompts do not fix this. The verification gate must live somewhere the executing agent cannot modify.
What I do now
Two structural gates, and the first matters more than the second.
1. Independent Validator with Provider Isolation
The model that executes a task never scores it. Where billing allows, the validator model sits behind a completely separate provider.
My early version of this rule only checked for a different model name (claude-3.5 vs claude-3.7). That quietly failed for months: both slugs were routed through the same API account and identical system-prompt wrappers. Two different model names, one single point of failure.
2. The Workspace Content-Hash Gate
To prevent an agent from doing nothing and pretending it succeeded, the orchestrator computes a content hash over the entire repository before and after the dispatch, strictly excluding the task’s own report directory. Writing a markdown report can no longer masquerade as doing engineering work.
Credit belongs to community member saltexx, who suggested using isolated git worktrees for differential comparison before the current content-hash gate was built. His worktree approach is even more rigorous than content hashing, and it correctly identified that detecting a no-op is an operating system and filesystem question, not an LLM judgment call.
What is still broken
We should not pretend this is fully solved. While dogfooding the harness, I found two of my own validation gates broken in both directions: one passing a task cell that never ran because a CLI exit code returned 0 on an API error, and another failing a task that had completed cleanly because of unanchored string matching on the word “error”.
The tool built to enforce accountability was itself flawed, and nothing was watching it.
I packaged these safeguards into wb-flow. It enforces a strict three-gate verdict contract (Infra, Artifact, and Oracle) and isolates Worker and Validator roles across different models. You can test it in any repo:
npx wb-flow
Enter fullscreen mode Exit fullscreen mode
If you are running coding agents across complex repositories, how are you verifying that your agents actually changed the code rather than just writing reassuring prose?