An agent patch is a hypothesis about your codebase. It makes three claims: the invariants still hold, the output changes are intended, and the new failures are not flukes. CI should test all three claims before a human opens the diff.
The gate I keep coming back to is a three-checkpoint contract. Property delta. Fixture delta. Flake freeze. Each checkpoint answers a question a reviewer would ask anyway; the goal is to make the answers machine-checkable so the review budget goes where it matters.
Patch generation is getting cheaper, which moves the constraint. Give a patch generator free model access and a free server option — MonkeyCode is the concrete case this article is written around — and the cost per patch attempt drops toward zero. Review time does not drop with it. If reviewing one patch costs more than generating ten, the agent wins by attrition.
Disclosure: This article was prepared as part of MonkeyCode’s product outreach.
The contract below is model-agnostic. It does not care whether the patch came from a free endpoint, a local model, or a teammate, and treating the free tier as an availability claim rather than a benchmark is enough. Nothing here depends on a model name, a quota, or an uptime number.
Checkpoint 1: property delta
A property test describes an invariant. An LRU cache never exceeds its capacity. A token bucket never goes negative. A serializer round-trips. Checkpoint 1 runs the property suite on the merge-base, runs it again on the patch, and diffs the results.
Rule: any property that passed on baseline and fails on HEAD blocks the merge. The patch did not “change behavior” — it broke a standing contract with the rest of the code.
from hypothesis import given, strategies as st
@given(
st.integers(min_value=2, max_value=64),
st.lists(
st.tuples(st.sampled_from(["get", "put"]), st.text(min_size=1)),
min_size=50,
max_size=300,
),
)
def test_lru_size_never_exceeds_capacity(capacity, ops):
cache = LRUCache(capacity)
for action, key in ops:
if action == "put":
cache.put(key, len(key))
else:
cache.get(key)
assert len(cache) <= capacity
Enter fullscreen mode Exit fullscreen mode
Both runs emit JUnit XML. A short parser compares them:
import sys
import xml.etree.ElementTree as ET
def failures(path):
root = ET.parse(path).getroot()
return {
case.get("name"): int(case.get("failures", "0"))
for case in root.iter("testcase")
}
baseline, head = failures("baseline.xml"), failures("head.xml")
broken = [
name for name, fails in baseline.items()
if fails == 0 and head.get(name, 1) != 0
]
if broken:
print("contract violation:", broken)
sys.exit(1)
Enter fullscreen mode Exit fullscreen mode
One rule is easy to miss. If the agent adds a new property, that property should fail on baseline code. A property that passes on both old and new code proves nothing; it is a test that never learned to say no.
Checkpoint 2: fixture delta
Property tests are generative. Fixtures are evidence. A fixture is a fixed input with a locked output, and it catches what properties rarely do: deliberate shifts in format, ordering, or rounding that still respect the invariants.
Rule: every changed fixture must be named in FIXTURES.md, in the same commit that touches it, with a one-line justification. The checkpoint lists the changed files and greps the manifest.
CHANGED=$(git diff --name-only "$(git merge-base HEAD origin/main)" HEAD -- tests/fixtures/)
for fixture in $CHANGED; do
grep -qF "$fixture" FIXTURES.md || { echo "unannounced fixture: $fixture"; exit 2; }
done
Enter fullscreen mode Exit fullscreen mode
An agent that reformats a golden file to match its own style has not fixed the test; it has moved the finish line. The manifest forces intent to be stated in the same commit, where a reviewer can actually see it.
Checkpoint 3: flake freeze
A flaky test makes every other verdict radioactive. One nondeterministic failure in the property suite poisons the delta; one in fixtures hides real change. The freeze is simple: a test that fails even once during a gate run is quarantined, and a non-empty quarantine blocks the merge. The agent cannot retry into a green run, because the retry loop belongs to the human, and this loop is closed.
rm -f .quarantine/current
for run in 1 2 3; do
if ! pytest tests/flaky -q; then
echo "failure on run $run" >> .quarantine/current
fi
done
[ -s .quarantine/current ] && { echo "FROZEN: quarantine must stay empty for 3 runs"; exit 3; }
Enter fullscreen mode Exit fullscreen mode
Three consecutive clean runs empty the quarantine. The rule sounds harsh. It is the only way to keep the first two checkpoints honest.
The contract, assembled
The full gate is a twenty-line script plus the JUnit diff. CI runs it on every patch the agent proposes:
#!/usr/bin/env bash
set -euo pipefail
BASE=origin/main
MERGE_BASE=$(git merge-base HEAD "$BASE")
WORK=/tmp/contract-base
trap 'git worktree remove --force "$WORK" 2>/dev/null' EXIT
# Checkpoint 1: property delta
git worktree add -f "$WORK" "$MERGE_BASE"
(cd "$WORK" && pytest tests/property -q --junitxml=baseline.xml)
pytest tests/property -q --junitxml=head.xml
python compare_junit.py "$WORK/baseline.xml" head.xml || exit 1
# Checkpoint 2: fixture delta
for fixture in $(git diff --name-only "$MERGE_BASE" HEAD -- tests/fixtures/); do
grep -qF "$fixture" FIXTURES.md || { echo "unannounced fixture: $fixture"; exit 2; }
done
# Checkpoint 3: flake freeze
rm -f .quarantine/current
for run in 1 2 3; do
if ! pytest tests/flaky -q; then
echo "failure on run $run" >> .quarantine/current
fi
done
[ -s .quarantine/current ] && { echo "FROZEN"; exit 3; }
echo "contract PASS"
Enter fullscreen mode Exit fullscreen mode
Reading the verdict
Exit Meaning Action 0 contract PASS merge candidate 1 invariant broken reject; return the failing property 2 behavior changed silently reject; require a manifest line 3 suite unstable freeze; wait for three clean runsEach failure mode tells the agent what to do next. That is the property of a good gate: it rejects, and it explains.
Who should not use this
The contract inherits the quality of your properties. A suite of tautological tests produces a gate that says yes to everything, so the real work is writing invariants that can fail. A team starting from zero does not need a contract yet; it needs one or two properties that matter.
Golden fixtures rot in UI-heavy and integration-heavy codebases, where output changes with every dependency bump. There, Checkpoint 2 generates noise instead of signal, and FIXTURES.md becomes a lie by week two.
The freeze makes a stable suite a precondition. A codebase with dozens of flaky tests stays frozen forever, and the team disables the gate out of frustration. Stabilize the suite before installing the freeze.
Finally, this contract does not approve security-sensitive changes. A patch that touches authentication or crypto still needs a human who understands both the patch and the threat model. No generator, free or paid, replaces that.
The bottom line
Every developer is now a reviewer, and the pipeline that feeds reviewers is what needs testing. The three-checkpoint contract is that test: it checks the patch, and it checks the suite that checks the patch.
Start with Checkpoint 1 alone. One invariant, one diff, one week of patches — the rest of the contract will suggest itself.