The problem: AI outputs are probabilistic, and prompts have a ceiling
LLMs produce probabilistic outputs. No matter how good your prompt is, edge cases will fail — hallucinations, omissions, format drift, and confident-sounding rationalizations that don’t hold up.
I noticed this while using Claude Code daily: the AI would say “done” but the file wasn’t written. It would claim “logs updated” but the timestamps were three days old. The AI wasn’t lying — probabilistic output is inherently unstable.
Pure prompt engineering is fighting probability with probability. The ultimate defense must be deterministic, mechanical checks.
The solution: 4 of 5 steps are scripts. Only 1 requires AI.
I built an agent configuration system with a closed-loop feedback mechanism:
self-model.md (current self-cognition)
↓
Session executes (AI works based on config)
↓
Growth data accumulates (what worked, what failed)
↓
quality-gate.py detects staleness (file timestamps + exit codes, pure Python)
↓
Writes .self-model-stale flag to disk
↓
Next startup: health-check.py detects flag → triggers AI to regenerate self-model
↓ (loop closes)
Enter fullscreen mode Exit fullscreen mode
4 steps are mechanical scripts: file timestamp checks, exit code gates, JSONL audit trails, flag file I/O.
1 step requires AI: content regeneration — synthesizing accumulated growth data into updated self-cognition.
Machines do the checking. Humans and AI do the judging. This isn’t philosophy — it’s engineering.
Key design decisions
1. Zero dependencies, stdlib only
Every script uses only Python’s standard library. A quality check tool can’t introduce new dependency risks.
2. Dual-layer gate: soft reminder + hard block
- Process layer (soft): Rule execution rate low? Remind, but don’t block.
- Output layer (hard): Learning logs not updated? Exit 2, hard block. Delivery must be complete.
The boundary isn’t importance — it’s “can this be fixed later?”
3. Filesystem as database
No vector databases. No cloud services. All identity data, growth logs, and audit records are local Markdown + JSON files. Git-auditable, offline-capable, fully self-sovereign.
External validation: submitting to a 100K-star project
I extracted one module (delivery-gate) from my personal system and submitted it to ECC (228k stars).
Result: maintainer daltino reviewed and approved it with praise. Maintainer affaan-m personally merged two follow-up PRs. A 200-line Python script went through 4 rounds of community bot review + human maintainer review, catching 9 issues I hadn’t found in self-testing.
Open-source community review is the best free QA you’ll ever get. This became my “open-source flywheel” methodology: build for yourself → extract module → find a community gap → submit PR → merge back into your own system.
If you want to do something similar
- Dogfood it first. My system ran through 50+ real sessions before I submitted anything.
- Scripts, not prompts. If you can check it with an if/else in Python, don’t describe it in natural language.
- Small PRs win. For large projects, 100-300 lines is the sweet spot for maintainer review.
- Use the gap-filling template. “This repo has X and Y. But there is no Z. This PR fills that gap.”
The real takeaway
I’m a junior-year undergrad. My raw coding speed probably doesn’t beat CS majors who eat LeetCode for breakfast. But I’ve learned one thing that matters more:
The core competency of the AI era isn’t typing speed — it’s knowing what to let AI do, and what must be enforced with deterministic rules.
Related: Causal swap experiment (n=30) confirms that config rules measurably shape agent behavior. WITH rule: 73% vs WITHOUT: 20%. Fisher’s exact p=0.0092. Full paper | DEV.to post
Update July 8, 2026: The causal swap experiment has been expanded to n=30 with statistically significant results (p=0.0092). Scoring protocol validated via independent blind rating (n=8, 87.5% agreement). Full paper.
🤖 Fact-checked 2026-07-10: GitHub PR status verified against API.
中文版：掘金/YuhaoLin2005yhl · Code on GitHub
답글 남기기