분산 스토리지 개론: 작동 방식 및 실제로 필요할 때

Distributed storage isn’t magic — it’s a set of trade-offs. Here’s how it actually works under the hood.

Key Stats

Metric Figure Orgs running distributed storage (mid-size+) ~68% Primary driver: HA/failure tolerance 74% Single-node disk failure recovery time 2–8 hours Teams that regretted distributing too early ~23%

What Makes Storage “Distributed”

Single-node: One process, one machine. If it dies → outage.

Distributed: Data spread across multiple nodes. Individual failures don’t cause data loss or downtime.

Single Node:   App → [Server + Disk] → if dies → GONE
Distributed:   App → [Node A] [Node B] [Node C] → if A dies → B+C serve

Enter fullscreen mode Exit fullscreen mode

The Three Problems It Solves

1. Availability

Hardware fails. Distributed systems survive individual component failures:

Failure Type Single-Node Distributed (3+ nodes) Disk crash Outage Auto-rebalance, zero downtime Machine crash Outage Other nodes take over Rack power loss Outage If cross-rack → degraded but alive

Rule: >99.9% uptime SLA? Distributed gets you there by design.

2. Scale Beyond One Box

One machine has limits (~500TB raw in a big chassis). Need 1 PB+? Add nodes.

3. Geographic Distribution

Multi-region for compliance, latency, DR:

US-East (Nodes 1-4)  ← primary
EU-West (Nodes 5-7)   ← compliance + EU users
APAC   (Nodes 8-10)   ← Asia + DR target

Enter fullscreen mode Exit fullscreen mode

How Data Gets Placed: Consistent Hashing

Most systems (RustFS, Ceph, Cassandra) use consistent hashing:

1. Hash the object key → big integer
2. Map to position on a hash ring (circle of values)
3. Each node owns a range of the ring
4. Object stored on node(s) whose range contains its hash

Why not hash % num_nodes? Adding/removing a node with consistent hashing moves only ~1/N of data. Mod-N reshuffles everything.

Replication vs. Erasure Coding

Replication (Simple, Space-Heavy)

Store N complete copies on different nodes.

• 3× replication: 200% space overhead, tolerate 2 failures, fast reads
• Best for: hot data, frequent reads

Erasure Coding (Complex, Space-Efficient)

Split data into fragments + compute parity.

• 4+2 scheme: 50% overhead, tolerate 2 failures, slower recovery
• 8+3 scheme: 37.5% overhead, tolerate 3 failures
• Best for: warm/cold data, capacity-sensitive workloads

Factor 3× Replication 4+2 EC 8+3 EC Space overhead 200% 50% 37.5% Failures tolerated 2 2 3 Read speed Excellent Good OK Write speed Good Moderate Slower

Most production systems use both: replicate hot data, erasure-code cold.

Consistency Models

Strong Consistency (CP)

Every read returns the most recent write. Always.

• Cost: Higher latency during writes.
• Used by: RustFS (default), Ceph, financial systems.

Eventual Consistency (AP)

Writes acknowledged immediately; reads might be briefly stale.

• Benefit: Lower write latency, higher availability.
• Used by: S3 (cross-region), Cassandra, DynamoDB.
• Acceptable for: Content delivery, analytics, logs.

Practical reality: Most systems are strong-consistency within a DC, eventual across regions.

CAP Theorem Explained

CAP = Consistency + Availability + Partition tolerance. Pick two.

But P is not optional — networks partition. Real choice:

CP AP During network split Stop writes to avoid divergence Accept writes, reconcile later Data safety ✅ No corruption ⚠️ Possible conflicts Uptime ⚠️ Partially unavailable ✅ Fully available Examples RustFS, Ceph, MongoDB Cassandra, DynamoDB, S3 x-repl

Most object storage is CP — divergent data is worse than brief unavailability.

When You DON’T Need Distributed Storage

Real costs to consider:

Cost Reality Complexity Monitoring 3–15 nodes vs 1 Latency Network round-trips vs local disk Overhead Metadata services consume CPU/RAM Learning curve 3–6 months to proficiency Debugging More moving parts = harder troubleshooting

Stay single-node when: <10TB, downtime acceptable, no ops team, dev/homelab, budget tight.

RustFS, MinIO, SeaweedFS all run great as single-node. Distribute when you hit a concrete wall.

Where Distributed Storage Breaks

1. Split-brain: Both sides of a partition accept writes → divergent data. CP systems refuse minority-side writes; AP systems hope for the best.
2. Rebalancing storms: Adding a node triggers data migration that can degrade performance. Schedule during low traffic.
3. Slow node problem: One slow node gets more retries → more load → slower. Thundering herd cascade.
4. Multi-node failure: Losing 3 nodes simultaneously (rack failure) may require human intervention.
5. Monitoring complexity: “Is the cluster healthy?” has 6+ dimensions, not a yes/no answer.

From Single Node to Cluster: Progressive Phases

Phase 1: Single Node (now) — official command, sourced verbatim from RustFS GitHub README [NOT EXECUTED IN CI]
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Phase 2: Replicated Cluster (need HA)
  Add --cluster-nodes flag, deploy on 3 machines

Phase 3: Erasure-Coded Cluster (need efficiency)
  Configure ec.scheme=4+2 (needs 6+ drives)

Phase 4: Multi-Region (need geo-redundancy)
  Add remote nodes, configure async replication rules

Enter fullscreen mode Exit fullscreen mode

Don’t skip phases. Teams jumping from Phase 1→4 usually spend the next quarter debugging.

Bottom Line

Distributed storage solves real problems at the cost of real complexity. The right move for most teams: start single-node, distribute when you hit a concrete wall.

RustFS runs as a trivially simple single-node deploy and scales to clustered erasure-coded with the same binary — making it a strong “start here, stay here” option for teams leaving MinIO or outgrowing standalone setups.

FAQ

What’s the difference between distributed storage and just putting files on a server?
Single-server storage has a single point of failure. If that disk dies or that machine loses power, your data is gone until you restore from backup. Distributed storage spreads data across multiple nodes so individual failures don’t cause data loss or downtime. The trade-off is operational complexity — you’re now managing a cluster instead of a filesystem.

When should I use distributed storage vs. single-node?
Use single-node (RustFS standalone, MinIO single-drive, SeaweedFS) for <10TB, non-critical workloads, homelabs, or development environments. Add distribution when you need high availability, capacity beyond one machine, throughput beyond one NIC/disk, or geographic redundancy. Don’t distribute for distribution’s sake — it adds latency, complexity, and operational overhead.

How does erasure coding work in simple terms?
Erasure coding splits data into N fragments, computes M parity fragments, and stores them across different drives/nodes. You can lose any M fragments and still reconstruct the data. A 4+2 scheme stores 4 data + 2 parity chunks across 6 locations; you lose any 2 and recover. Compared to 3x replication, 4+2 uses only 1.5x raw space while tolerating the same number of failures. The cost: reconstruction is CPU-intensive.

What’s CAP theorem and why does everyone argue about it?
CAP says a distributed system can guarantee at most two of three: Consistency, Availability, Partition tolerance. “P” isn’t optional — networks do partition — so the real choice is CP (consistency over availability during a split) or AP (availability, accept temporary inconsistency). Most storage systems are CP because divergent data is worse than briefly unavailable data.

Is RustFS distributed? How does it compare to Ceph?
RustFS supports both single-node and clustered deployment. In clustered mode it uses erasure coding with configurable schemes (default 4+2) and supports adding/removing nodes dynamically. Ceph is more mature for massive deployments (100+ nodes, petabyte-scale) with unified object/block/file storage via RADOS. RustFS is simpler to operate, has lower resource overhead (95MB idle vs Ceph’s GB-range), and is written in Rust which eliminates a class of memory-safety bugs. For most teams under 50PB, RustFS hits the complexity/performance sweet spot.

Feedback? GitHub Issues