Most people describe S3 as a place to put files, and that description is exactly why the interesting engineering is invisible to them. S3 is not a filesystem, it is not a database, and it is not a bigger disk. It is a durability machine dressed up as a key-value store, and the number that defines it, eleven nines of durability, is not marketing. It is a hard engineering claim that shapes every layer of the system. Eleven nines means the design is built so that if you store ten million objects, you should expect to lose a single one of them roughly once every ten thousand years. You do not reach a number like that by copying data three times and hoping. You reach it by treating loss as a physics problem, measuring the failure rates of real hardware, and engineering the redundancy, the placement, and the repair speed until the arithmetic gives you the nines you promised.
This piece works S3 the way I would want a staff engineer to work it in a design review. We start from the requirements and the scale numbers, because the durability target and the object counts decide the architecture before any component does. Then we build the pieces: the flat keyspace that separates object storage from files, the write path that refuses to acknowledge until data is durable, the read path, and the two genuinely hard subproblems that make S3 what it is. The first is durability itself, which means erasure coding, independent placement across fault domains, and a repair loop that never stops. The second is the keymap, the index that maps trillions of keys to their physical fragments while staying strongly consistent. We finish with the failure mode that quietly eats nines and the patterns that carry to other systems. One note on numbers before we start. Every failure rate, latency, and storage figure here is an industry-typical estimate meant to drive the sizing and the reasoning, not a measured production metric from any specific service.
Requirements: durability is the product
The functional surface of S3 is small enough to state in a breath. You store an object, which is an opaque blob of bytes, under a key inside a bucket. You retrieve that object by key and get the exact bytes back. You list keys by prefix and you delete objects. There are refinements on top, versioning, multipart upload for large objects, lifecycle rules, and access control, but the core is those few verbs against a flat namespace.
The functional list is a handful of verbs. The non-functional column, led by eleven nines of durability, is where the entire design actually lives.
The design does not live in that verb list. It lives in the non-functional column, and durability sits at the top of it. Durability is the probability that an object you successfully stored is still intact and retrievable at some later time, and S3 targets 99.999999999 percent of it per object per year. That is a different property from availability, which is the probability that the service can serve your request right now, and which S3 Standard targets at a more ordinary 99.99 percent. A system can be briefly unavailable and lose nothing, which is annoying but recoverable. A system that loses your bytes has failed at the one job object storage exists to do, and there is no recovery from that. The rest of the requirements follow: effectively unbounded scale into the trillions of objects and exabytes of capacity, individual objects up to five terabytes, and predictable performance that does not degrade as the store grows.
Durability is not a feature bolted onto object storage. It is the entire product, and every architectural decision in S3 is subordinate to keeping your bytes exactly as you left them.
The numbers that set the scale
Before choosing any mechanism, put the scale on the table, because the size of S3 rules out whole categories of design. Amazon has reported that S3 holds well over four hundred trillion objects and averages on the order of one hundred fifty million requests per second across the service. Those two numbers alone kill any design that keeps a global lock, scans a central table, or coordinates across the whole fleet on the hot path.
Eleven nines is a statement about expected loss over time, and the raw scale, trillions of objects and exabytes of bytes, rules out any centralized or coordinated design.
Translate the durability target into something concrete. Eleven nines per object per year means an expected annual loss rate of one object in one hundred billion. Store ten million objects and the expectation is one loss every ten thousand years. Store a hundred billion objects and you expect to lose about one per year, which is why a service holding four hundred trillion objects has to keep pushing the number higher and repairing faster than hardware fails. On the physical side, disks are unreliable in bulk. A single drive has an annualized failure rate somewhere around one to two percent, so in a fleet of millions of drives, thousands are failing every year as a matter of routine. Storage overhead matters at this scale too, because the difference between copying data three times and erasure coding it at roughly 1.4 times its size is the difference between paying for three exabytes and paying for 1.4 exabytes to protect one. The scale math points in one direction. You cannot centralize anything, you cannot afford naive replication, and you must assume hardware is failing continuously rather than occasionally.
The high-level architecture
With the target and the scale named, the shape of the system follows. A request from a client, whether a PUT or a GET, arrives at a fleet of stateless front-end servers behind load balancers. Those servers authenticate and authorize the request, then split the work across three planes that each do one job and scale independently.
Stateless front-end servers sit above three independent planes: a metadata index that maps keys to locations, a storage plane that holds erasure-coded fragments, and a durability engine that repairs in the background.
The metadata plane, which I will call the keymap, maps a key to the set of physical locations where that object’s fragments live. It is a large distributed index, partitioned by key so that no single machine owns more than a slice of the namespace. The storage plane is the fleet of storage nodes and their disks, which hold the actual bytes, encoded into fragments for redundancy. The durability engine is a set of background processes that continuously audit stored data, detect drives and fragments that have gone bad, and rebuild lost redundancy before it can accumulate into loss. That third plane is the one most designs forget and the one that actually produces the nines. The organizing principle is the same one that shows up in every system that scales cleanly. Keep the request path short, partition state so nothing is global, and run the expensive correctness work, in this case durability repair, continuously in the background rather than on the request.
Object storage is a flat keyspace, not a filesystem
Be precise about what an object is, because the model is the reason S3 scales the way it does. Object storage is not a hierarchy of directories. It is a flat map from a string key to an immutable blob of bytes plus its metadata. The slashes you see in a key like reports/2026/q1.pdf are just characters in the string. There are no real folders, and listing a prefix is a range scan over sorted keys, not a walk down a directory tree.
Block storage hands out raw addressable blocks, file storage imposes a directory tree and byte-range updates, and object storage maps a flat key to an immutable whole object.
That flatness is a deliberate trade against the two storage models it sits beside. Block storage, the kind an EBS volume or a raw disk gives you, exposes fixed-size numbered blocks and lets a filesystem on top impose whatever structure it wants, with fast in-place updates to any block. File storage, the kind NFS gives you, exposes a POSIX tree with directories, permissions, and the ability to modify a byte range in place. Object storage gives up both in-place updates and the directory tree. Objects are written and read whole, and an overwrite replaces the entire object with a new version rather than editing bytes inside it. In exchange, the model has no tree to lock, no inode hierarchy to keep consistent, and no cross-object relationships to maintain, which is exactly what lets the keyspace partition flatly across thousands of machines and grow without bound. The immutability of an object is not a limitation to work around. It is the property that makes durable, distributed, versioned storage tractable.
The write path: durable before acknowledged
The single most important rule in the write path is that S3 does not tell you a PUT succeeded until the data is durably stored across multiple independent failure domains. The acknowledgment is a promise, and the system earns the right to make it before it answers.
A PUT is not acknowledged until the object has been checksummed, erasure coded, and written across at least three Availability Zones, and only then is the keymap updated.
Follow a PUT through. The front-end server receives the bytes and computes a checksum so it can verify integrity end to end. For a small object the server erasure codes the data into fragments; for a large object the client uses multipart upload, sending parts in parallel that are each handled the same way and stitched together at completion. The fragments are then written to storage nodes spread across multiple Availability Zones, which are physically separate data centers with independent power and networking. The write is considered durable only once enough fragments have been committed to disk across those zones that the object could survive the loss of an entire Availability Zone plus additional device failures. Only after that durability commit does the front-end update the keymap to point the key at its new fragment locations, and only then does it return a 200 to the client. If the client got an error or a timeout instead, it must assume the write did not happen and retry, because S3 will never have acknowledged a write it did not make durable. That ordering, data durable first and index updated second, is what makes the eleven nines promise honest rather than optimistic.
The read path: index, then fetch
A GET is the inverse and is shorter, because the hard work was done at write time. The read path is a lookup followed by a fetch, with reconstruction only when it is needed.
A GET resolves the key to fragment locations in the keymap, fetches the data fragments, and reconstructs the object only if a fragment is missing.
The front-end server takes the key, looks it up in the keymap, and gets back the set of locations holding the object’s fragments. It reads the data fragments from the storage nodes and streams the bytes back to the client. When every data fragment is healthy and present, no decoding is required, the fragments are simply the object’s bytes in order, and the read is close to a direct disk read plus network transfer. When a data fragment is missing because a drive is dead or slow, the server reads enough of the surviving fragments, including parity, to reconstruct the missing piece on the fly using the erasure code, then serves the reconstructed bytes. This is the same degraded-read mechanism that lets the system tolerate failures without the client ever noticing, and it runs on the read path only when a fragment is unavailable, which is the uncommon case. Hot objects are usually fronted by a content delivery network such as CloudFront, so the most-requested bytes are served from an edge cache and never touch the storage plane at all. The read path stays fast because reconstruction is the exception and the common case is a location lookup and a fragment read.
Replication versus erasure coding
The center of the durability design is how you store data redundantly, and the choice between full replication and erasure coding is the trade that everything else rests on. Both survive failures. They pay very different prices to do it.
Triple replication tolerates two lost copies at three times the storage cost; a 10-of-14 erasure code tolerates four lost fragments at 1.4 times the cost.
Full replication is the simple answer. Keep three complete copies of every object on three independent machines, and you can lose any two and still serve the object. It is easy to reason about and cheap to read from, since any single copy is a complete object. The cost is that you pay three times the storage for one times the data, and at exabyte scale that multiplier is enormous. Erasure coding buys the same or better fault tolerance for far less overhead. You split an object into k data fragments, compute m parity fragments using a Reed-Solomon code, and store all n equal to k plus m fragments on different machines. Any k of the n fragments are sufficient to reconstruct the whole object, so the scheme tolerates the loss of any m fragments. A common configuration is k equals 10 and m equals 4, which stores fourteen fragments, tolerates the loss of any four, and costs only 1.4 times the original size rather than three times. That is a durability level comparable to or better than triple replication at less than half the storage cost, which is why large object stores erasure code the bulk of their data. The trade you accept is complexity, a read that needs reconstruction must gather k fragments and do math, and small objects can be inefficient to code, so tiny objects are sometimes replicated or packed together instead.
Reed-Solomon, worked through
Erasure coding is worth understanding concretely rather than treating as a black box, because the guarantee it gives is exact and the intuition is simple. A Reed-Solomon code treats the object’s data as points that define a polynomial, then evaluates that polynomial at extra positions to produce parity. Any sufficient subset of the points recovers the polynomial, and from it the original data.
A 10-of-14 code splits the object into ten data fragments and computes four parity fragments; any ten of the fourteen are enough to rebuild the object exactly.
Walk the 10-of-14 example. The object is divided into ten equal data fragments, call them d1 through d10. The encoder computes four parity fragments, p1 through p4, where each parity fragment is a specific linear combination of all ten data fragments over a finite field. All fourteen fragments are the same size, one tenth of the object each, so the total stored is fourteen tenths, the 1.4 times overhead. The code is used in systematic form, which means the ten data fragments are the object’s own bytes rather than encoded symbols, and that is exactly why a healthy read needs no decoding at all and only a degraded read pays for the math. Now suppose four of the fourteen fragments are lost, in any combination of data and parity. Because you still hold ten fragments, and any ten fragments carry enough independent equations to solve for the ten original data values, the decoder reconstructs every lost fragment exactly. Lose a fifth fragment before repair and the object is gone, which is why m equals 4 is a deliberate choice about how much simultaneous failure you are willing to survive. The elegance is that the guarantee is not probabilistic within the tolerance. Any four losses are always recoverable, not usually recoverable. That hard boundary is what lets you compute durability precisely instead of estimating it.
Where eleven nines actually comes from
Eleven nines is not produced by the erasure code alone. It is the product of three factors working together, and removing any one of them collapses the number. Understanding this is the difference between reciting that S3 is durable and knowing why.
Durability is the product of three factors: how many simultaneous losses you tolerate, how independent the fragments are, and how fast repair shrinks the danger window.
The first factor is redundancy, the m in the code, which sets how many fragments can fail before data is lost. More parity means more nines, but with diminishing returns and rising cost. The second factor is independence. The durability arithmetic assumes fragment failures are uncorrelated, which is only true if the fragments live in genuinely independent failure domains, different disks, different racks, different power circuits, different Availability Zones. If two fragments share a fault, one event takes both, and your fourteen supposedly independent fragments behave like fewer, which silently destroys nines. The third factor, and the one people underestimate, is repair speed. Durability depends far more on how quickly you rebuild lost redundancy than on how much redundancy you start with, because what actually causes loss is a burst of failures landing within the same repair window. If a drive dies and you rebuild its fragments in hours, the probability that four more independent failures hit the same stripe inside that short window is vanishingly small. Let repair take weeks and that probability climbs. The formula intuition is that durability rises sharply as the ratio of mean time between failures to mean time to repair grows, so cutting repair time is the cheapest way to add nines. S3 gets its eleven nines by combining enough parity, strict independence of placement, and a repair loop fast enough to keep the danger window tiny, all at once.
Independent placement across fault domains
Independence is not free and it is not automatic. It is the direct result of a placement policy that spreads an object’s fragments so that no single physical failure can take more of them than the code can survive.
Fragments are spread across Availability Zones and racks so that losing an entire zone removes fewer fragments than the code can tolerate, keeping the object readable.
The rule the placement layer enforces is that no fault domain holds more than m of the n fragments. Concretely, with a 10-of-14 code you spread the fourteen fragments across several Availability Zones so that no single zone holds more than four of them, and the loss of any one entire zone then removes at most four fragments, leaving at least ten, exactly enough to reconstruct. The same logic applies at finer grain inside a zone, across racks, power distribution units, and individual storage nodes, so that no rack failure and no single node failure removes more than the tolerance allows. This is why the write path insists on committing across at least three Availability Zones before acknowledging, and it is why One Zone storage classes, which keep all fragments in a single Availability Zone, offer the same per-object durability against device failure but explicitly give up the protection against a full zone loss. Placement is where the abstract independence assumption in the durability math becomes a concrete constraint on which disks a fragment is allowed to land on. Get placement wrong and the erasure code still runs, the storage overhead is still paid, and the durability is quietly far worse than the number on the page.
The keymap: an index at trillions of keys
The second genuinely hard subproblem is the index. Every GET and every PUT begins with a lookup or an update in the keymap, the structure that maps a key to its object’s fragment locations, and that structure has to hold trillions of entries and serve them at a hundred million requests per second without a hot spot bringing it down.
The keymap stores one record per object version, keyed by bucket and key, and is partitioned by key range so the namespace spreads across thousands of independent shards.
Each keymap record holds the bucket and key, a version identifier, the size and checksum of the object, and the list of fragment locations that make up the object. The index is partitioned by key range, so a contiguous slice of the sorted keyspace lives on one shard and the full namespace spreads across thousands of shards that each own a manageable piece. Because keys are sorted, listing a prefix is a range scan within a shard or a small number of adjacent shards rather than a global operation. The hard part is skew. A workload that writes keys with a common sequential prefix, timestamps or incrementing ids, concentrates all its traffic on one shard and throttles, which is the origin of the old advice to randomize the leading characters of your keys. Modern S3 largely removes that footgun by automatically splitting a hot partition into more shards as its request rate climbs, so throughput scales with the number of prefixes in use, and the published guidance of thousands of requests per second per prefix is a floor that grows automatically. The keymap is the part of S3 that most resembles a distributed database, and it earns its complexity because a flat trillion-key namespace with strong consistency is a genuinely hard thing to index.
Strong consistency without a tax
For its first fourteen years S3 was only eventually consistent for overwrites and deletes, which meant a GET right after a PUT could return the old bytes for a while. In December 2020 S3 became strongly consistent for every operation, read-after-write for new objects, overwrites, deletes, and even list operations, and it did so without adding cost, latency, or reducing availability. That combination is the interesting part.
Strong consistency requires that once a write updates the keymap, no later read can be served a stale cached location; a witness protocol invalidates caches at commit time.
The consistency problem in S3 is not in the object bytes, which are immutable, but in the keymap. When you overwrite an object, the write commits new fragments and updates the key to point at them. The danger is a cache somewhere in the read path still holding the old location, so a subsequent read follows the stale pointer to the previous version. Eventual consistency tolerated that window; strong consistency forbids it. The way to close the window without slowing every read is a lightweight persistence and witness layer that tracks, for each key, which version is current, and guarantees that a read either sees the effect of the latest committed write or is redirected to confirm the current location before it answers. The metadata subsystem commits the new mapping to a strongly consistent store and ensures cached locations that could be stale are invalidated or revalidated at read time, so the read barrier is paid only when there is actually a risk of staleness rather than on every request. The lesson worth taking is that strong consistency did not require throwing out caching or accepting a latency penalty. It required identifying the exact place staleness could enter, the cached key-to-location mapping, and placing a precise, cheap barrier there instead of a broad, expensive one everywhere.
The durability flywheel: audit, scrub, repair
Redundancy at write time is necessary but not sufficient, because redundancy decays. Drives die, bits rot silently, and every failure that goes unrepaired brings an object one step closer to loss. The mechanism that actually sustains eleven nines is a background loop that never stops running.
A continuous loop audits fragments against their checksums, detects loss or corruption, rebuilds missing fragments from the survivors, and verifies that full redundancy is restored.
The loop has four stages that feed each other. First, continuous auditing reads stored fragments and verifies them against their stored checksums, which catches silent corruption, the bit rot where a disk returns wrong bytes without reporting an error, as well as fragments lost to dead drives. Second, detection flags any fragment that is missing, unreadable, or fails its checksum, and identifies every object stripe whose redundancy has dropped below full. Third, repair reconstructs the missing or corrupted fragments from the surviving ones using the erasure code and writes fresh fragments onto healthy, independent hardware, restoring the stripe to its full fourteen fragments. Fourth, verification confirms the rebuilt stripe is complete and correctly placed, then the loop returns to auditing. The speed of this loop is the repair time that dominates the durability arithmetic, so it is engineered to run constantly and in parallel across the fleet rather than reactively. This flywheel is why durability is a running property, not a one-time write-time guarantee. An object stored once and never re-examined would slowly rot toward loss; an object continuously audited and repaired stays at full redundancy indefinitely.
The failure mode that eats nines: correlation
The failure that actually destroys durability is rarely the one the arithmetic worries about. The math assumes independent failures, and independence is exactly what a real data center conspires to violate. Correlation is the quiet enemy of nines.
Failures nest inside each other. A disk takes one fragment, a rack takes everything sharing its power and switch, and the placement rule is that no ring may hold more fragments than the code can lose.
Consider what the eleven nines calculation depends on. It assumes that the probability of losing five fragments of a fourteen-fragment stripe within a repair window is the product of five small independent probabilities, which is astronomically tiny. Now introduce correlation. If several of an object’s fragments happen to share a rack, and that rack loses power, you lose them together as a single event, and the failures were never independent. The same happens with a shared network switch, a bad firmware version rolled out to a batch of drives at once, or a cooling failure that overheats a room. Each of these turns what the model counted as many independent low-probability events into one event that removes many fragments simultaneously, which is precisely the scenario that overwhelms the code’s tolerance. This is why placement across independent fault domains is not a nice-to-have but the load-bearing assumption of the whole durability claim, and why operational discipline, staged rollouts of firmware and software, correlated-failure testing, and monitoring for placement that has drifted out of independence, matters as much as the code parameters. The number on the page is only real if the failures stay independent, and keeping them independent is continuous operational work, not a one-time design decision.
The trade-offs, and what transfers
Every choice in S3 bought something and cost something, and naming both sides is what separates understanding the system from admiring it. Erasure coding bought durability at 1.4 times storage instead of three, but cost the complexity of reconstruction on degraded reads and inefficiency on tiny objects. Committing writes across three Availability Zones before acknowledging bought zone-loss survival, but cost write latency, since the slowest of three independent data centers gates every PUT. Strong consistency bought a simpler mental model for every application built on S3, but required a witness layer that had to be engineered to add no latency to the common read. The flat immutable object model bought effortless horizontal scale, but gave up in-place updates and any notion of a real directory tree. In each case the design traded flexibility for a property it valued more, durability or scale or predictability, which is the right trade for a system whose job is to hold your bytes forever and do it billions of times a second.
Four decisions from S3, engineer durability as arithmetic, erasure code over replicate, place for independence, and repair continuously, transfer to any system that must not lose data.
The reason S3 rewards this much study is that its hardest lessons are not about object storage. Four of them transfer directly to any system that must not lose data, and you will build several of those. First, treat durability as arithmetic you can compute, not a feeling you can add, and drive it by the three levers that actually move it: redundancy, independence, and repair speed. Second, prefer erasure coding to replication once storage cost matters, because you buy equal or better fault tolerance for a fraction of the space. Third, remember that redundancy is only as good as its independence, so place your copies or fragments across genuinely separate failure domains and treat correlation as the real threat. Fourth, understand that durability is a running property maintained by continuous audit and repair, not a one-time write-time act, because unrepaired redundancy decays toward loss. Learn to see which of these levers your own storage bottleneck sits on and S3 stops being a magic bucket and becomes a method you can apply.
I teach system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.
Read the free lessons: https://systemdesign.academy
답글 남기기