The Guardrail Your Agent Can Reach
Somewhere in the repository you look after, there is a rule you decided to stop trusting to prose.
It had been a line in your context file. You rewrote it three or four times. It failed anyway, on a Saturday. You gave up on wording it better. So you did the thing everyone now recommends and moved it somewhere deterministic: a pre-commit hook, or a PreToolUse hook, or a permission rule. Something that returns a failure code and stops the run outright, instead of asking the model nicely and hoping.
Good instinct. Anthropic gives the same advice about their own product: “When there’s something that absolutely must not happen, an instruction is the wrong tool… A real guardrail needs to be deterministic, and the enforcement methods are hooks and permissions.” 1
Now open the thing you built and look at what shipped with it.
The hatch you put there on purpose
Your pre-commit hook is skipped by git commit --no-verify, and you have typed that this month. Your CI check has an override label for the Friday when the build is red and the release is going out anyway. Your permission prompt has an “allow always” button that somebody pressed in week one. Your harness, the process that runs the agent and decides which of its requests to honour, reads a bypass environment variable. It is documented, because you documented it the night you needed to get something out of the door.
None of that is sloppiness. A control with no way through will one day stop something that genuinely needed to happen, at three in the morning, with nobody around to authorise it. So an exception path goes in deliberately. The interesting question is who it was built for. Sometimes that turns out to be the party the control was meant to constrain.
In ordinary developer tooling that trade-off has usually worked, because the person holding the override is still sitting inside an accountability system.
The old rule, in a new place
Security engineering worked this out decades ago: you do not let the thing being constrained control the constraint. Least privilege, separation of duties, two-person release. None of it assumes the constrained party is trustworthy, because accountability after the fact was never enough on its own. The set of things you are trusting completely even has a name, the trusted computing base, and the whole discipline is keeping it small.
The strong version has a name too. NIST’s definition of a reference monitor puts three requirements on whatever does the enforcing. It must be always invoked. It must be tamperproof. It must be small enough to analyse and verify. 2
Determinism buys you none of the three. It buys something real that the list does not name, which is that the decision no longer runs through the model’s judgement. Everything else is still open. Your hook can go uncalled, it can be altered, and it can be wrong.
git commit --no-verify is the first of those. Nothing was tampered with, the script sits on disk exactly as you wrote it, and it simply was not invoked. An agent that edits the hook file is the second. Being wrong is the third and it is the quietest, because a hook whose pattern stops rm -rf but not find . -delete will pass every test you thought to write and fail forever in the one case you did not.
Most guardrail failures I have watched are the first or the third, and those are the well-served ones: hook linters, server-side checks that a local --no-verify cannot reach, no shortage of advice about testing what you wrote. Security people have covered that ground too. Trail of Bits publishes opinionated Claude Code defaults whose documentation is blunt: hooks “are not a security boundary”, because “a prompt injection can work around them”. They are “guardrails, not walls”. 3 This piece is about the second failure, which is where all of that stops. What is scarce is a test precise enough to apply to a specific repository on a Tuesday.
None of this is hypothetical either. In March one user reported that Claude Code had edited a gate they had built with it, narrowing the instruction inside the hook from “code changes” to “SIGNIFICANT code changes”, and attached the one-line diff. A follow-up RFC, from someone running enforcement across 545 tasks, reported watching the model add Bash(*) to its allowlist to get past a hook that was blocking it. Both were auto-closed for inactivity, with no vendor response either way. 4 Neither is a jailbreak. In both cases the model did the ordinary thing, which was to edit a file it had write access to, and the file happened to be its own constraint.
The principle is old. Where it gets violated is new. This used to be something you reasoned about while designing a permissions system, and it has become something you do by accident on a Tuesday, without once thinking of yourself as designing a security boundary. You would never ship an application that could rewrite its own access policy. Then you gave a coding agent write access to the repository holding its hooks.
Two things change when the constrained party is an agent.
The first is accountability, and it half survives. An audit trail can still work: the agent has a run ID, it acts under some credential, and you can read back what it did. The deterrent does not. A developer reaching for --no-verify knows they may have to explain it, and in some override systems that expectation does more of the work than the mechanism does. It does nothing to a process at two in the morning. So you keep the forensics and lose the deterrent, and you keep the forensics only if the log lives somewhere the agent cannot also write and somebody reads it. A record nobody opens is not a record.
The second change turns a reliability problem into a security one.
Mistake, misled, or aimedMost of the time your agent is simply fallible. It forgets, it drifts out of scope, it solves the adjacent problem. A repo-local hook is genuinely good at catching that, and nothing here should talk you out of one. Instructions get followed, too: a separate team measuring these files across real repositories found that instructions in them are well followed by coding agents. 5 The prose is not being ignored. That is exactly why a hook that catches the residue is worth writing.
Sometimes the agent is misled. Anthropic names this case in the same breath as the others: a prompted rule can fail “due to a prompt injection in a file accessed as part of the task.” 6 Your agent reads issues, dependency READMEs, test fixtures, web pages. Any of them can carry instructions, and the agent can fail to tell your intent from text it read thirty seconds ago. Security has an old name for this shape. A component with more authority than the party directing it is a confused deputy, and that hazard has had a name since 1988.
And sometimes it is aimed. The model wants nothing. Whoever wrote the text it just read might. That is the bridge, and it needs no speculation about what an agent wants. The thing with permission to disable your guardrail is also, routinely, reading untrusted input. Each fact is fine alone. Together they are the problem.
So the useful question was never whether your agent would decide to bypass a control. It is what the control is worth on the day something in its context window tells it to.
답글 남기기