A backup can be perfectly readable and still fail to bring a service back.
The database may restore while the application remains offline because a signing key is missing, an object-storage bucket was not included, DNS points at the wrong place, or an identity provider cannot be reached. A recovery document may describe every command correctly but omit the order in which dependencies must return. None of this is visible when the only test is whether a backup job completed.
That gap became unusually concrete this week. Risky Business reported that an attacker breached Romania’s cadastral agency, wiped systems and backups after a failed extortion attempt, and left official services offline for a week. The report also noted that the agency appeared to retain an offline copy.
The interesting product question is therefore not simply, “Do we have a backup?” It is: “Can this team use what it has to restore a working service under realistic failure conditions?”
A recovery drill that is allowed to break things
Imagine connecting backup locations, a database inventory, service dependencies, and the recovery runbook for a small containerized application. The product creates a minimally runnable copy in an isolated environment. It never targets production.
A drill then applies a specific destructive scenario to that copy:
- delete the primary database;
- invalidate one set of credentials;
- remove a host or storage dependency;
- start recovery from a selected historical point.
The important constraint is that the system follows the team’s existing runbook exactly. It should not silently repair missing steps, because the missing steps are what the drill is trying to reveal.
The result is more useful than a green or red backup status. It records the recoverable data point, actual recovery time, and first failed step. If the database returns but the application page does not, the report should connect that failure to the missing dependency and the relevant runbook section.
This direction is not starting from zero. Veeam’s SureBackup can start machines from backups in an isolated environment and run verification tests. AWS Elastic Disaster Recovery also documents non-disruptive recovery drills. The narrower opportunity is to make the runbook itself the test subject and produce an actionable dependency-gap report for smaller teams.
A deliberately small first version
I would start with one database, object storage, and a containerized application. The team supplies a recovery document and selects a scenario. The drill runner provisions an isolated network, restores the components, executes only the documented steps, and captures logs and timing.
The first report needs to answer four questions:
- What point in time was actually recoverable?
- How long did it take to reach a working business flow?
- What was the first missing or incorrect recovery step?
- Did any temporary resource attempt to reconnect to production?
That is enough to improve a runbook and repeat the same test next month. Broad backup-format support, automatic remediation, and production failover orchestration can wait.
The strongest case against it
An isolated environment may create false confidence. Production networking, identity systems, vendors, rate limits, and private keys are difficult to reproduce safely. A successful copy-level drill does not prove that a real incident will go smoothly.
There is also a security paradox: restoring an infected backup can recreate the attacker’s foothold. A drill platform would need strict egress controls, short-lived credentials, complete cleanup, and an explicit way to mark dependencies that could not safely be reproduced.
Finally, existing disaster-recovery vendors already cover part of this workflow. A new tool would need to remain focused on something they do not make simple enough: testing the team’s actual written procedure and showing the first business-level dependency that prevents recovery.
Would a runbook-driven destruction drill expose useful gaps in your stack, or would the isolated environment be too different from production to trust?
답글 남기기