Disaster Recovery for AI Systems

Disaster recovery for an AI system is unusual in one respect: several of its largest assets can be rebuilt from other assets. A vector index is derivable from documents. A fine-tuned model is derivable from a dataset and a training script. For those, the recovery objective is not a backup schedule — it is a rebuild time you can calculate today and shorten deliberately.

RTO and RPO, applied here

Two numbers, per asset, and the discipline is to write them down before designing anything.

Term Description RTO Recovery time objective: how long the thing may be unavailable. A business decision, and the one that decides how much you spend, since halving RTO is usually far more expensive than halving RPO. RPO Recovery point objective: how much recent data may be lost. Sets backup frequency and replication mode. An RPO of zero requires synchronous replication and therefore a latency cost on every write.

Set them per asset, not per system. A single system-wide RTO forces you to engineer everything to the strictest requirement, which is how DR budgets get spent on things nobody would have missed. Conversation history and the prompt registry do not need the same treatment.

One AI-specific addition to the usual pair: degraded-mode capability. For many AI features the honest recovery plan is not full restoration but a smaller model, a cached answer or a non-AI fallback, available within minutes, while full capability returns in hours. Write that down as a third target; it is often the one that actually protects users.

The asset inventory

The deliverable of a DR exercise is this table, filled in for your own system. The classification column is the part that matters.

Asset Description Prompts and templates Derivable from version control, which is already replicated. RPO effectively zero, RTO minutes. The failure mode is prompts that live in a database rather than in git — if so, fix that first, it is the cheapest DR improvement available. Routing and model configuration Same as prompts: it should be in version control. If the current production configuration cannot be reconstructed from a repository, you have a DR gap that no backup fixes. Source documents / corpus Usually irreplaceable if it is customer data, and usually already the system of record with its own backup regime. Everything else in the retrieval stack is derived from this, so it is the asset to protect hardest. Embeddings and vector index Derivable from the corpus. Rebuild cost is compute and time, not information. See the next section — this is the asset whose RTO is a calculation. Fine-tuned weights Derivable from the training dataset, the script and the base model — at a cost. Whether to back up the weights or plan to retrain is an arithmetic decision, and storing a 40 GB artefact is almost always cheaper than a retraining run. Training datasets Irreplaceable, and frequently the least well protected thing in the system, because it lives on a workstation or in a notebook environment. If the dataset is gone, the fine-tune is gone permanently. Conversation history Irreplaceable and usually the primary product data. Standard database DR applies: point-in-time recovery, tested restores, an RPO measured in minutes. Evaluation sets and results Irreplaceable in practice — an eval set represents accumulated judgement that would take months to rebuild. Small, so protect it like source code, in git. Provider credentials Not recoverable, but re-creatable. The DR requirement is the ability to mint and distribute a new key quickly, which is exactly the rotation procedure, so a tested rotation is also a tested DR control. Audit logs Irreplaceable, and often subject to a retention obligation that survives the incident. Write-once storage, replicated, separate from the primary system.

Derivable assets: RTO is arithmetic

For anything in the derivable class, do not guess the recovery time — compute it. Start with the vector index, which is the common case.

Rebuild time for a retrieval index, all assumptions labelled.

  D    documents in the corpus ................. 500,000
  C    mean chunks per document ................    12
  N    total chunks = D × C .....................  6,000,000
  E    embedding throughput, chunks/second ......   400   (measure yours: it
                                                          depends on model,
                                                          batch size, hardware
                                                          or provider rate limit)
  I    index build throughput, vectors/second ...  20,000

  embedding time = N / E = 6,000,000 / 400   = 15,000 s = 4.2 hours
  index build    = N / I = 6,000,000 / 20,000 =    300 s = 5 minutes
  extraction and chunking (usually I/O bound)  ≈ 1–2 hours
  ------------------------------------------------------------------
  total rebuild ................................ roughly 6 hours

So the RTO for this index is 6 hours unless you do something about it. Two
levers, and both are ordinary engineering rather than DR spending:

  1. Parallelism. Embedding is embarrassingly parallel. Ten workers gives
     E = 4,000 and the embedding stage falls to 25 minutes — provided your
     rate limit or GPU pool actually supports ten workers. Check that; the
     limit is usually not the code.

  2. Back up the embeddings rather than only the index. Embeddings are the
     expensive stage; the index build is five minutes. Storing the vectors
     turns a 6-hour RTO into a 30-minute one.

     Storage cost of that backup:
       6,000,000 vectors × 1,024 dimensions × 4 bytes = 24.6 GB
     which is trivially cheap against 4.2 hours of embedding compute
     repeated at every recovery, and cheap against 6 hours of degraded
     retrieval for users.

Enter fullscreen mode Exit fullscreen mode

The same reasoning applies to fine-tuned weights. Retraining is derivable; the question is whether the retraining cost exceeds the storage cost, and it always does. Store every promoted checkpoint with its digest and its run metadata, as model weights in CI/CD describes, and keep the dataset that produced it in the same protected place — a checkpoint whose dataset is lost cannot be reproduced or improved, only used.

Re-embedding has a correctness trap as well as a time cost. If the embedding model has changed or been updated since the index was built, a rebuild produces vectors in a different space from any that survived, and mixing them silently degrades retrieval. Record the embedding model identifier and version alongside the index, and treat a mismatch as a full rebuild — re-embedding migrations covers the procedure.

What is genuinely irreplaceable

Four things, and they deserve the whole DR budget because everything else can be rebuilt from them.

1. The corpus and the conversation history. Standard database and object-storage DR: point-in-time recovery, cross-region replication of backups, immutability so that a deletion — malicious or accidental — does not replicate. The single most common real disaster is not a region failing; it is a delete that replicated perfectly.
2. Training and evaluation datasets. Move them out of personal environments into versioned storage with the same protection as production data. This is usually an organisational fix rather than a technical one.
3. Audit logs. Write-once, replicated, retention enforced by the storage rather than by a policy document.
4. The ability to authenticate to your providers. Not the key itself but the capability: an account you can still log into from a second location, with a second administrator, and a documented procedure for minting a replacement key. An organisation that loses access to its provider account has an outage that no backup addresses.

The provider is part of your DR plan

If your system depends on a hosted model, that provider’s availability is inside your RTO whether you planned it or not. Three questions, answered in advance:

• What happens during their outage? Failover to a second provider is the answer, and it only works if the second integration exists, is credentialed, and is exercised — which is what chaos testing a provider outage is for. An untested fallback is a plan, not a capability.
• What happens if they retire the model? Deprecation is more likely than an outage and less often planned for. Your eval set is what makes a forced migration a measurable change rather than a leap, and having a second model already validated turns it into a configuration change.
• What happens if the relationship ends? Account suspension, a pricing change you cannot accept, a contract dispute. The mitigation is portability: prompts that are not tuned to one model’s quirks, an abstraction over the provider, and outputs validated against a schema rather than against one model’s habits. Provider-agnostic code and prompt portability are the practices.

Note that self-hosting does not remove this class of risk, it exchanges it. Your provider becomes your hardware supply, your data centre and your own operations team — which are more within your control and not automatically more reliable.

A restore you have not tested is a hope

The finding that recurs in every DR exercise is that the backups existed and the restore did not work: an incomplete schema, a missing index definition, an encryption key held only in the environment being restored, a procedure that assumed a service that is also down.

1. Restore to a scratch environment quarterly. Time it. The measured time, not the intended one, is your RTO — and almost always exceeds the estimate by a factor that surprises people.
2. Rebuild the index from the corpus once, for real. The arithmetic above gives an estimate; running it once gives a fact, and usually reveals a rate limit or a pipeline step nobody remembered.
3. Verify content, not just completion. Run the eval set against the restored system. A restore that completes and retrieves the wrong documents has failed, and only an eval catches that.
4. Test the credential path. Can you mint a new provider key without the person on holiday? Can you decrypt the backup without the key manager that is also in the failed region? These questions have embarrassing answers the first time.
5. Record the actual numbers and update the plan with them. A DR document with aspirational numbers is worse than one with honest, disappointing numbers, because someone will make a promise based on it.

Then take the measured numbers to the capacity review, where the question of whether they are still acceptable belongs — the quarterly capacity review has a slot for exactly this.

Related

• Multi-Region AI Serving
• The Quarterly Capacity Review