Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.
AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md
When I started this project, I thought the star of the show would be the optimizer.
The part that reads failures, rewrites the prompt, and gets smarter over time.
That is still the part people notice first.
By the time v0.3.0 shipped, it was no longer the part I trusted most.
That honor went to the gate.
A Self-Editing System Can Quietly Destroy Its Own Baseline
That is the problem in one sentence.
If you let a system change the prompt that shapes its own future behavior, then a bad edit is not just a one-off bug. It becomes part of the baseline. It gets inherited by every later run.
That is why I stopped thinking about the gate as a feature. It is the blast door.
If the blast door is weak, the rest of the system becomes dangerous no matter how clever the optimizer looks.
What the Gate Actually Does in v0.3.0
By v0.3.0, the promotion gate had seven deterministic checks:
- sample floor
- effect size
- confidence threshold
- frozen sections
- edit distance
- drift detection
- Oracle Drift Guard
No LLM decides whether an edit gets promoted.
That was a deliberate line in the sand. If an LLM judges its own edits, the whole system starts grading its own homework. I wanted the final promotion boundary to be code, not vibes.
The Numbers That Made Me Trust It
The best numbers in v0.3.0 were not about improvement. They were about restraint.
- 0% false positives across the shipped field-test evidence
- 8/8 adversarial edits blocked
- 0 false negatives observed in adversarial validation
- no bad promotions in local synthetic, cloud synthetic, or Docker-backed flows
If a self-editing system is going to fail while it learns, this is how I want it to fail.
Conservatively. Verbosely. With receipts.
Oracle Drift Guard Ended Up Mattering More Than I Expected
This was one of the quieter v0.3.0 features, but I think it is one of the most important.
Oracle Drift Guard exists for a nasty kind of failure where the optimizer, the scorer, and the benchmark all share the same wrong idea of success.
That is more dangerous than an obvious bug because the whole system can look internally consistent while still drifting in the wrong direction.
The prompt starts optimizing for the wrong thing. The scorer rewards it. The corpus keeps reinforcing it. Everything lines up and says “looks good.”
That is not learning. That is coordinated self-deception.
I think more AI systems need explicit defenses against that class of failure.
The Safety Story Was Bigger Than Statistics
People hear “gate” and think p-values. Fair enough. The confidence threshold matters.
But the real safety story in v0.3.0 was wider than that.
materialize_candidate_prompt() replaced raw str.replace(), which means missing old_text fails loudly instead of pretending an edit happened.
Frozen sections draw a hard line around prompt content the analyzer should never touch automatically.
Edit distance limits keep the system from calling a wholesale rewrite “one edit.”
Rollback with lineage means a promoted change is not just stored. It is traceable and reversible.
That is what I mean when I say the gate became the product boundary. It is not one check. It is a safety stack.
The Optimizer Still Didn’t Win, and That Made the Gate More Valuable
The optimizer still did not produce a promotable edit in v0.3.0.
The best cloud candidate from mistralai/mistral-small-3.2-24b-instruct showed weak positive movement, but nowhere near enough confidence.
The local 4B analyzer mostly produced null edits.
The first separated-role run produced zero proposals.
That can sound disappointing if your only success metric is promotion.
It looks very different if your first success metric is “do not let the system quietly corrupt its own baseline while the optimizer is still immature.”
On that metric, the gate absolutely earned its keep.
The Surprise Was How Hard Safe Failure Actually Is
I used to think safety would be the boring part. Add thresholds. Add checks. Move on.
That was naive.
The hard part was not inventing names like confidence or drift. The hard part was making sure the system measured the right thing, at the right point in the flow, against the right baseline prompt, with enough attached evidence that a rejection meant something concrete.
Once a system edits itself, every shortcut gets more expensive.
That was one of the main lessons of v0.3.0.
What I Learned
The optimizer is replaceable. The gate is not. If I swap in a stronger analyzer tomorrow, the gate still has to be the thing I trust more than the model.
I also learned that statistical rigor is necessary but not sufficient. P-values and effect sizes matter, but so do provenance, materialization correctness, adversarial testing, rollback, and drift checks. Safety is a stack, not a number.
I also came away thinking the gate has become a better diagnostic tool than I expected. It is no longer just a blocker. In v0.3.0, it is separating different failure classes: null edits, locally plausible net-zero edits, and weak-positive but underpowered edits. That makes the gate more useful than a binary “promote/reject” label suggests.
And I learned the hardest thing emotionally: a system that refuses to move can still be healthy. Sometimes a long string of rejections is not proof that the project is broken. It is proof that the project still has a boundary.
Why Developers Should Care
If you are building any loop where an AI system can revise its own behavior, ask this before you ask anything else:
what stops the system from rewarding its own mistakes?
If the answer is basically “another LLM will tell us if it looks good,” you do not have a safety story yet.
I started this project thinking the smartest part would matter most.
v0.3.0 taught me the opposite. The part that says no is the reason the rest of the system gets to keep trying.
If you were building a self-modifying system, how conservative would you make the safety boundary before the product became useless? Would you rather ship a system that almost never promotes, or one that moves faster but occasionally corrupts its own baseline? Where do you draw that line?