What Nobody Tells You About Running 9 Autonomous Agents on a Real Gym
We spent 107 days waiting for AI agents to hallucinate. The port proxy failed silently for 19 days instead.
Here’s the honest list of problems that surprised us the most.
The Failure Modes We Designed For (But Didn’t Get)
Before we launched our 9-agent system in a physical fitness studio, we spent weeks on threat modeling:
- Agent jailbreaks → Constitution-based governance with 11 production constraints
- Hallucinations → Cross-validation gates between Stella (audit) and Momo (operations)
- Model drift → Version-pinned inference and hash-constrained outputs
- API outages → Graceful degradation with fallback providers
None of these happened. Not once in 107 days.
The Failure Modes That Actually Hit Us
1. A Port Proxy Went Silent for 19 Days
This is the one we talk about most because it’s the scariest. A cleanup script had a regex edge case — it reported “clean” every run but left the stale rule intact. No errors surfaced. Data was disappearing every minute.
The founder caught it during a routine infrastructure review. Within hours, the agents encoded it as ERR-001 — a permanent prevention rule in our constitution. It can never recur.
2. Memory Creep (6.8GB→13.4GB Over 60 Days)
Our 9 agents ran on 2 CPU cores and 3.6GB RAM. But memory crept upward as conversation histories accumulated. By day 60, we were at 13.4GB and climbing.
The fix wasn’t a better model or more RAM. It was a staggered restart schedule — agents take turns cycling while the system stays online. Simple. Structural. Permanent.
3. RSS Lock After Gateway Restart
The swap file was still declining after a Gateway restart. Stella caught the pattern and correctly triaged “wait, don’t force.” The system recovered on its own within 2 hours.
A human operator would have force-restarted everything and lost the recovery context.
The Architecture Lesson
Our agents run on DeepSeek-V4 flash. Not GPT-4. Not Claude. A cheap, accessible model.
The immune system works because of the constitution — not because the models are smart.
Constitution > Prompts. Architecture > Model size. Verification > Intelligence.
If we had designed for smarter models instead of structural constraints, all three bugs above would still be undiscovered.
What Problem Surprised You?
This is the question we posted to the community. We’re genuinely curious — because real production experience doesn’t match what the benchmarks suggest.
→ Join the discussion: github.com/ZWISERFIT/ZWISERFIT/discussions/35
→ Full system: github.com/ZWISERFIT/ZWISERFIT
→ Production constraints: github.com/ZWISERFIT/retroonto
답글 남기기