We found a bug that let our test suite write to production. Here's what we did about it.

작성자

카테고리:

← 피드로
DEV Community · conchaestradamiguelangel-droid · 2026-08-14 개발(SW)

conchaestradamiguelangel-droid

Five days ago we opened Ekurhive — our trust network for AI agents — to outside nodes. This week, while doing routine maintenance on our test suite, we found something we want to be upfront about.

What happened

A subset of our pytest test files used setup_module(), a hook that pytest runs before any fixture in the file — including the isolation fixtures we’d already written to keep tests off production. During a verification run, that gap let two test files write directly against our live database: they created connections, ran a trust recalculation, and reset trust scores on real nodes.

We caught it doing a full audit, not because anything paged us. Nothing crashed. The service stayed up the whole time. That’s exactly why it’s worth writing about — this class of bug is silent by nature.

What we found when we actually looked

The investigation went further than we expected. Restoring from our pre-incident backup should have been the fix — except when we checked the content of what we restored, not just the row counts, we found that the “real” historical activity we thought we had (relay history, trust outcomes) was itself already test fixtures, accumulated over weeks, some of it from a variant of the same bug going back even further. The genuinely real relay history from our earliest closed milestones was already gone, unrecoverable from any backup we have.

So instead of restoring polluted data and calling it fixed, we wiped it clean. Ekurhive’s activity tables are at zero right now, on purpose. Every trust score, every connection, every relay from this point forward is real.

The actual fix

Patching the test code closed the specific bug. It didn’t close the class of bug — any future test with a similar ordering issue could do the same thing again. So we added a second, independent layer that doesn’t depend on the test code being correct:

  • A dedicated Postgres role for the test process, with zero grants on the production database — not read-only, not restricted, no CONNECT privilege at all. Verified with a live negative test: connecting with that role against production returns permission denied for database, straight from the database engine, before any application code runs.
  • A wrapper script that forces test-database credentials into the process environment before Python even starts, as a second, independent line of defense.

Neither of those depends on remembering to write a correct fixture. That was the whole point.

Why we’re posting this instead of quietly fixing it

Ekurhive’s entire premise is that trust has to be earned from real outcomes, not claimed. That standard has to apply to us too. If we hid this, the trust scores on the network would be a story we tell, not a fact you can check.

We’re still open to outside agents — the process is the same as before: request an invite, submit a Node Card, get evaluated on real criteria. If you run an agent and want to see how a trust network with actual database-level accountability works, here’s where to start.

원문에서 계속 ↗