Why Your Elasticsearch Backups Might Fail When You Need Them Most

Why Your Elasticsearch Backups Might Fail When You Need Them Most: Lessons from the July 2026 Recovery

In the world of Site Reliability Engineering, there is a painful, often career-defining difference between having a backup and having a recovery. This reality hit our team with visceral clarity during a high-stakes production recovery event in July 2026.

While we believed our standard operating procedures were robust, the technical nuances of Elasticsearch 7.17.x proved otherwise. Our previous SOP didn’t just have “gaps”—it allowed for a “Partial” success state that nearly resulted in total data loss.

The following insights are derived from the hard-won lessons of that recovery. We have since scrapped the old methods and codified these findings into our formal SOP (Version 1.0). If you are responsible for Elasticsearch clusters, these changes are the difference between a clean restoration and a catastrophic failure.

The “Partial” Snapshot is a Silent Failure:

The most dangerous trap in Elasticsearch administration is accepting a snapshot that isn’t perfect. In July 2026, we learned that a snapshot can complete with a state of PARTIAL, a “silent failure” that often occurs due to node shutdowns, shard unavailability, or filesystem issues on the NFS server. A partial snapshot means some shards were simply skipped, leaving invisible data holes that you won’t discover until your application is back online and throwing errors.

Our updated SOP now strictly forbids proceeding with a restoration if the state is anything other than SUCCESS. If you encounter a PARTIAL state, do not attempt to overwrite or restart the full process immediately. Instead, you must resolve the underlying infrastructure issue and take a NEW, dedicated snapshot specifically for the missing indices.

“A backup is considered successful when: Snapshot state is SUCCESS.”

The Global State Bloat:
When executing a backup, the default behaviour is often to capture the cluster’s global state. While this seems comprehensive, it is an architectural poison when restoring to a different environment. During our recovery, we moved data from our source cluster (Primary Node) to a target cluster (Restore Cluster). Including the global state risked “poisoning” the new cluster with shard allocation awareness attributes and persistent tasks that were specific to the old hardware.

By explicitly setting “include_global_state”: false, we ensure the backup is truly portable.

PUT /snapshot/es_backup/all_indices
{
“indices”:”*”,
“include_global_state”:false
}

This prevents the target cluster from trying to mimic the source’s runtime configuration, which is essential for a clean start on the Restore Cluster cluster.

Don’t Restore the “Ghost” Indices:

A common reflex during a disaster is to restore everything inside the snapshot. However, our recovery post-mortem identified several “ghost” indices—system indices like .tasks and .geoip_databases—that should be ignored.

These are runtime indices that Elasticsearch 7.17.x recreates automatically. Attempting to force a restore of .tasks, for example, can cause the target cluster to hang or crash as it tries to resume background tasks that were specific to the source cluster’s state.

Always exclude these system indices to allow the target cluster’s native operations to initialize correctly.

The “Green” Health Mandate

A successful recovery begins long before the PUT command is run; it starts with the health of the source. One of our key lessons from July 2026 was that snapshots taken during node restarts or maintenance windows are inherently unreliable.

We now mandate a “Status Green” precondition. If your cluster is “Yellow,” you have unassigned or initializing shards. Taking a snapshot of a degraded cluster is effectively building on shifting sand—it is the primary recipe for the PARTIAL failures mentioned earlier. Before you touch the backup API, verify that there are no relocating shards and the cluster is stable.

Metadata is the True Backup

A snapshot file residing on the NFS server is useless if the cluster that created it is gone and you have no record of what was inside. During the heat of a recovery, you cannot rely on the Elasticsearch API to tell you what you have if the cluster itself is down.

We now require engineers to archive snapshot metadata—JSON output detailing index counts, shard status, and failure logs—off-cluster. This metadata should be stored in a version-controlled repository or a secondary secure bucket, not just on the cluster being backed up.

GET /_snapshot/es_backup/?pretty

“Archive snapshot metadata (indices, failures, timestamps, shard status) along with operational records for future audits and disaster recovery.”
This allows you to verify that the document counts on your new Restore Cluster exactly match what was captured from the Primary Node source, providing the empirical proof required for a successful sign-off.
The Quarterly Integrity Test

The “set it and forget it” mentality is the enemy of Site Reliability. You do not have a backup until you have proven you can restore it. Our SOP now mandates a test restore on a non-production cluster under three specific triggers:

• Quarterly: A routine scheduled validation.
• Post-Upgrade: Immediately following any Elasticsearch version change.
• Infrastructure Change: Whenever there are modifications to the NFS backup server or network pathing.

These tests verify more than just data; they validate your Recovery Time Objective (RTO) and ensure that your application remains compatible with the restored mappings and settings.

Conclusion: The Future of Resilience

The transition to Version 1.0 of our Elasticsearch SOP marks our shift from “checking a box” to building true operational resilience. By enforcing “Green” state preconditions, excluding dangerous global state, and rigorously archiving metadata off-cluster, we have replaced “Partial” hope with technical certainty.

The July 2026 recovery was a wake-up call that forced us to evolve. If your primary cluster vanished today, would your metadata stored off the Backup server be enough to rebuild, or are you currently relying on a “Partial” hope?