I love reading technology history because it explains two powerful things: why a tool was born, and why people eventually moved on.
Data formats tell this story perfectly. None of them were random. Each one answered a real pain of its era then later met new pressures: bigger scale, faster systems, better developer experience, lower cost.
So instead of treating XML, JSON, and others like a feature checklist, let’s read them like chapters in a long engineering story.
1960s–1970s: CSV the practical beginning
CSV feels almost too simple to be important, but that simplicity is exactly why it survived. In the early decades of business software, teams mostly needed to move rows and columns between systems without ceremony. CSV became the universal handshake: boring, reliable, and easy for both humans and spreadsheets.
But that same simplicity became its ceiling. Once systems needed nested objects, strict typing, or safer interoperability, CSV started to feel fragile. Even today, though, it refuses to die because for flat data exchange, it’s still one of the fastest ways to get work done.
1987: XDR when machines needed a shared binary language
As distributed systems matured, the challenge shifted from “how do humans read this?” to “how do different machines interpret this identically?” XDR answered that with a platform neutral binary representation, crucial in RPC heavy infrastructure at the time.
It wasn’t designed to be pretty or beginner friendly; it was designed to be dependable across architectures. Over time, newer formats offered better tooling and developer ergonomics, but XDR’s role in early networked system reliability is foundational.
1998: XML the age of structure, governance, and enterprise trust
XML arrived when the industry needed structure and contracts. It gave teams a way to define rich hierarchical data, validate it, and agree on strict schemas across organizations. For enterprises, governments, telecom, and finance, this was gold: predictable integrations mattered more than elegance.
The downside was obvious to anyone who wrote lots of it: verbosity. XML payloads were heavy, parsing was expensive, and developer experience was often painful compared to what came next. Still, replacing XML entirely never happened in many regulated and legacy-heavy ecosystems, it remains deeply embedded because strictness and compatibility still matter.
Early 2000s: JSON the web’s favorite language for data
If XML was formal wear, JSON was a hoodie and sneakers. It arrived at the perfect moment: the web needed lightweight client server communication, and JavaScript was everywhere. JSON felt natural, minimal, and fast enough, with almost zero friction for developers.
That ease made JSON dominant for APIs and it still is. But at very large scale, teams often feel its limits: verbose text payloads compared to binary formats, weaker schema discipline unless enforced externally, and conventions for types like timestamps/decimals that can drift between services. Even so, for public APIs and product iteration speed, JSON remains hard to beat.
2001 onward (broad adoption in 2010s): Protobuf performance with contracts
As microservices exploded, “human readable” stopped being the top priority for backend to backend communication. Protobuf won hearts by combining compact binary encoding, strong schemas, and safe evolution over time. It gave large teams something JSON alone struggles to guarantee: clear contracts that scale across languages and services.
The trade off is intentional: less readability, more tooling, more discipline. You don’t casually open a Protobuf payload in a text editor for quick debugging like JSON. But in return, you get speed, smaller payloads, and better long term service compatibility which is why high throughput systems love it.
2006: MessagePack JSON’s fast cousin
MessagePack took a straightforward bet: keep the JSON like data model developers already understand, but encode it efficiently in binary. That made it attractive for systems wanting better throughput without a complete mental model shift.
It never replaced JSON universally, mostly because JSON’s ecosystem gravity is enormous and readability still matters in many teams. But where bandwidth and speed matter more than eyeballing payloads, MessagePack continues to shine.
2009: Avro schema evolution for data pipelines
Avro grew with big data and event driven architectures. In long lived pipelines, the hardest problem isn’t serialization alone it’s evolving schemas without breaking producers and consumers over months and years. Avro made that lifecycle easier and became a natural fit in Kafka and Hadoop style ecosystems.
Compared with Protobuf, Avro often feels more data platform centric than RPC centric. That distinction matters: teams choose based on where the center of gravity is service contracts vs analytical/event pipelines.
2013: Parquet and ORC analytics changed the rules
Row formats are fine for transactional reads/writes, but analytics workloads ask different questions: scan huge datasets, touch only selected columns, aggregate fast, and reduce storage cost. Parquet and ORC answered this with columnar storage and strong compression, massively improving query efficiency.
This wasn’t just an incremental upgrade it changed data economics. Warehouses and lakehouses became far more practical at scale. These formats didn’t replace JSON/XML in APIs; they replaced inefficient storage choices in analytical systems.
2014: CBOR compact structure for constrained environments
CBOR brought a JSON like structure into a standardized compact binary form, especially useful for IoT and constrained networks. It’s a reminder that format choices are often shaped by physical limits: device memory, network cost, battery life.
CBOR is not trying to be the default web API format; it wins where efficiency and compactness matter more than developer facing readability.
2015: FlatBuffers and Cap’n Proto chasing ultra low latency
Some domains (games, real time systems, high frequency services) can’t afford serialization overhead. FlatBuffers and Cap’n Proto were designed for near zero copy access patterns, letting systems read structured binary data with minimal transformation cost.
They are powerful but specialized. For many business APIs, that complexity isn’t worth it. For performance critical engines, it absolutely is.
Late 2010s+: TOML configuration with less pain
TOML rose because developers wanted config files that are readable and predictable without YAML’s sharp edges. It found a strong home in modern tooling ecosystems, especially where clean project configuration matters.
It’s not trying to be your universal wire protocol. It succeeds by doing one thing well: human friendly, dependable configuration.
2020s: Table formats over files Delta, Iceberg, Hudi
Modern data platforms learned that choosing Parquet alone isn’t enough. Teams also need transactional reliability, schema evolution, partition intelligence, and time travel on object storage. Table formats like Delta, Iceberg, and Hudi add that metadata/transaction layer.
This is an important shift: the conversation moved from “which file format?” to “which table abstraction over files?” a sign of platform maturity.
The pattern behind all replacements
When you zoom out, replacement is rarely about one format being “bad.” It’s about context changing faster than assumptions.
- CSV still wins for quick flat exports.
- XML still wins where strict governance and legacy contracts dominate.
- JSON still wins for public web APIs and developer speed.
- Protobuf/Avro win when scale and schema evolution become existential.
- Parquet/ORC win when analytics cost and scan speed define success.
Formats don’t really die. They find their level.
Final thought
The best engineers don’t just ask, “What’s modern?”
They ask, “What problem was this designed to solve, and do I have that problem?”
That’s why technology history is so useful: it helps us choose tools with clarity, not hype.
답글 남기기