Introduction
When a data platform grows from a few gigabytes to terabytes or petabytes, storing the data is only one part of the problem.
A typical data lake can store huge amounts of data cheaply in systems such as Amazon S3 or HDFS. The real challenge is making that collection of files behave like a reliable analytical table.
Consider an e-commerce company receiving millions of orders every day. Its analytics team may want to answer questions such as:
- How much revenue was generated last month?
- Which products are performing best?
- Can yesterday’s data be corrected without affecting today’s queries?
- What did the table look like before a faulty data pipeline ran?
A basic file-based data lake does not automatically solve these problems.
This is where Apache Iceberg becomes interesting. Iceberg is an open table format designed for huge analytical datasets. It adds a structured metadata layer, snapshots, schema evolution, partition evolution, and reliable table commits on top of data-lake storage.
In this article, I will look at the problem from an engineering perspective: where Hive Metastore fits in, why traditional table management becomes difficult at scale, and how Iceberg changes the architecture.
1. The Problem With a Traditional Data Lake
A simple data lake may look like this:
Data Sources
|
v
+----------------------+
| Data Lake |
| S3 / HDFS |
+----------+-----------+
|
v
Parquet / ORC Files
|
v
Spark / Hive / Trino
Enter fullscreen mode Exit fullscreen mode
This architecture is excellent for inexpensive large-scale storage.
However, a directory containing thousands or millions of files is not automatically a database table.
An analytical engine needs additional information:
- What is the table schema?
- Which files belong to the table?
- How is the data partitioned?
- Which files should be scanned for a particular query?
- What happened during the last write?
- Can the previous version of the table be recovered?
This is where metadata becomes critical.
2. Where Hive Metastore Fits In
The Hive Metastore provides metadata about Hive tables and partitions. Query engines can use this metadata to understand the structure and location of data.
A simplified architecture is:
Query Engine
Spark / Hive / Trino
|
v
+----------------+
| Hive Metastore |
| Metadata |
+-------+--------+
|
v
Partitions / Directories
|
v
Parquet / ORC
|
v
S3 / HDFS
Enter fullscreen mode Exit fullscreen mode
For example, an e-commerce sales table might traditionally be organized as:
sales/
├── year=2025/
│ ├── month=11/
│ └── month=12/
└── year=2026/
├── month=01/
├── month=02/
└── month=03/
Enter fullscreen mode Exit fullscreen mode
The partition structure can help an engine avoid scanning unrelated data.
For example, if an analyst asks for January 2026 sales, the engine can use the partition information to reduce the amount of data it needs to scan.
However, as the number of partitions and files grows, table management becomes increasingly important.
3. Where the Traditional Approach Starts Getting Difficult
Partition Management
Traditional partitioning can tightly connect the logical table to its physical directory structure.
If the workload changes, the original partitioning strategy may no longer be ideal.
For example:
Initial design:
sales/
year=2026/
month=01/
month=02/
month=03/
Enter fullscreen mode Exit fullscreen mode
Later, the company may need much finer-grained access:
year/month/day
Enter fullscreen mode Exit fullscreen mode
Changing the physical layout can become an operational concern.
Schema Changes
Real datasets rarely remain static.
Suppose an original order table contains:
order_id
customer_id
product_id
amount
Enter fullscreen mode Exit fullscreen mode
Later, the business adds:
payment_method
Enter fullscreen mode Exit fullscreen mode
A modern analytical table should be able to evolve without requiring every historical data file to be rewritten.
Partial Writes and Consistency
Imagine a pipeline is supposed to add 500 new Parquet files.
If a failure occurs after 300 files are written, simply looking at the storage directory does not tell the query engine whether those 300 files represent a complete committed table update.
A table format therefore needs a reliable way to define:
“This is the exact version of the table that readers should see.”
Metadata Growth
At large scale, the problem is not only the size of the actual data.
A table may contain millions of files, and the system also needs efficient information about those files.
This makes metadata management a first-class engineering problem.
4. Enter Apache Iceberg
Apache Iceberg takes a different approach.
Instead of making the physical directory structure the main source of truth for the table, Iceberg maintains a structured metadata hierarchy that tracks the table state and the data files belonging to it.
A simplified architecture looks like this:
Query Engine
Spark / Trino / Flink
|
v
Iceberg Catalog
|
v
Table Metadata
|
v
Snapshot
|
v
Manifest List
|
v
Manifest Files
|
v
Parquet / ORC
|
v
S3 / HDFS
Enter fullscreen mode Exit fullscreen mode
The important idea is:
The logical table is separated from the physical organization of its data files.
Iceberg’s specification describes table state through metadata files, snapshots, manifest lists, and manifest files rather than relying only on directory listings.
5. Understanding Iceberg’s Metadata Hierarchy
This is the part that makes Iceberg particularly interesting from a Big Data Analytics perspective.
Think of the metadata hierarchy as a chain:
Table Metadata
|
v
Snapshot
|
v
Manifest List
|
v
Manifest Files
|
v
Data Files
|
v
Parquet / ORC
Enter fullscreen mode Exit fullscreen mode
Each layer has a different responsibility.
Table Metadata
The table metadata keeps track of important table information such as the schema, partition configuration, and snapshots.
It acts as the entry point for understanding the current state of the table.
Snapshot
A snapshot represents the state of the table at a particular point in time.
Conceptually:
Snapshot 1
|
v
Snapshot 2
|
v
Snapshot 3 <-- Current
Enter fullscreen mode Exit fullscreen mode
When a successful table change is committed, Iceberg creates a new table state.
This snapshot-based design enables capabilities such as time travel and rollback. :
Manifest List
A snapshot points to a manifest list.
The manifest list tells Iceberg which manifest files belong to that snapshot.
Manifest Files
Manifest files contain information about data files, including file paths, partition information, and statistics.
This metadata can help the query engine determine which files need to be considered for a query. :contentReference[oaicite:3]{index=3}
Data Files
Finally, the actual records are stored in data files such as:
Parquet
ORC
Avro
Enter fullscreen mode Exit fullscreen mode
These files remain in the underlying storage system such as S3 or HDFS.
6. A Real-World Example: E-Commerce Sales
Consider an online shopping platform that processes millions of orders.
Its Iceberg table might contain:
order_id
customer_id
product_id
sale_timestamp
amount
payment_method
Enter fullscreen mode Exit fullscreen mode
Suppose the analytics team runs:
SELECT
product_id,
SUM(amount) AS revenue
FROM sales
WHERE sale_timestamp >= '2026-01-01'
GROUP BY product_id;
Enter fullscreen mode Exit fullscreen mode
The query engine does not simply scan every file in the storage system.
Iceberg’s metadata can help identify the relevant files and avoid unnecessary work through partition and file-level information.
The logical query remains focused on business data:
sale_timestamp
Enter fullscreen mode Exit fullscreen mode
rather than requiring the analyst to manually understand the physical directory structure.
This separation between logical queries and physical layout is one of the important ideas behind Iceberg’s design.
7. Hidden Partitioning
One of Iceberg’s useful features is hidden partitioning.
In a traditional partitioned data lake, users may need to understand how data is physically partitioned.
For example:
year=2026/month=01/
Enter fullscreen mode Exit fullscreen mode
With Iceberg, partitioning is treated as a table configuration rather than something that users must directly encode into every query.
For example, an analyst can write:
SELECT *
FROM sales
WHERE sale_timestamp >= '2026-01-01';
Enter fullscreen mode Exit fullscreen mode
The table format can use its partition information and data statistics during planning.
This means the physical organization can change without forcing users to redesign their SQL queries around directory names.
Iceberg’s documentation describes this as hidden partitioning and partition evolution.
8. Schema Evolution
Data schemas change constantly in real-world systems.
Suppose our original table is:
order_id
customer_id
amount
Enter fullscreen mode Exit fullscreen mode
Later we add:
discount
Enter fullscreen mode Exit fullscreen mode
Instead of treating this as a completely new table, Iceberg supports controlled schema evolution.
For example:
ALTER TABLE sales
ADD COLUMN discount DOUBLE;
Enter fullscreen mode Exit fullscreen mode
Other supported evolution operations include adding, dropping, renaming, and reordering fields under Iceberg’s schema-evolution rules.
This is especially useful for long-lived analytical datasets where historical data should remain usable while the business schema changes.
9. Time Travel: Looking at an Older Table State
One of the most useful consequences of snapshots is time travel.
Imagine this sequence:
10:00 AM
Snapshot 101
|
v
12:00 PM
Snapshot 102
|
v
03:00 PM
Snapshot 103
Enter fullscreen mode Exit fullscreen mode
If a pipeline accidentally introduces incorrect data at 03:00 PM, the older snapshot still represents an earlier table state.
This is useful for:
- Debugging
- Auditing
- Reproducing analytical results
- Recovering from incorrect changes
The important concept is that Iceberg tracks table state through snapshots rather than treating the current directory contents as the only version of the table.
10. Traditional Hive-Style Tables vs Apache Iceberg
Feature Traditional Hive-style Approach Apache Iceberg Metadata Metastore + table/partition information Structured table metadata Physical layout Often closely tied to directories Separated from logical table Transactions More limited in file-based workflows Atomic table commits Schema evolution Can require operational work Designed for controlled evolution Partitioning Directory/partition oriented Hidden partitioning Partition evolution More difficult Supported Time travel Not a core table-format feature Snapshot based Large-scale metadata Can become challenging Metadata hierarchyThe goal is not to say that Hive Metastore is “bad.”
Hive and its Metastore solved an important problem: giving query engines a way to understand tables and partitions in a distributed data environment. The limitation appears when organizations need richer table semantics, evolving schemas, changing partitions, and reliable versioned table states at very large scale.
11. What Iceberg Does NOT Solve Automatically
It is tempting to think that adopting Iceberg removes every data-engineering problem. It does not.
There are still operational considerations.
Metadata Maintenance
Every write can create a new snapshot, so old snapshots and metadata eventually need maintenance.
Iceberg provides operations for expiring snapshots, removing old metadata, deleting orphan files, and compacting data files.
Catalog and Engine Compatibility
An Iceberg deployment still needs a catalog and compatible processing/query engines such as Spark, Flink, Trino, or others.
Migration
Moving an existing data lake to Iceberg requires planning.
Engineers need to consider:
- Existing data layout
- Catalog configuration
- Query engines
- Partition strategy
- Data quality
- Migration and rollback plans
So Iceberg is not a magic switch. It is a table-format layer that addresses specific reliability and scalability problems.
12. When Should You Consider Iceberg?
Iceberg becomes particularly attractive when a data platform has:
- Large analytical datasets
- Object-storage-based data lakes
- Frequently changing schemas
- Multiple processing or query engines
- Large numbers of files
- Changing partition requirements
- Need for historical table versions
- Requirements for reliable table updates
For a small and mostly static dataset, a simple file-based solution may be enough.
The value of Iceberg becomes clearer when data volume, query complexity, schema changes, and operational requirements increase together.
13. Key Takeaways
The main lesson is that a data lake is more than a collection of files.
At small scale, directories and a metastore may appear sufficient. At larger scale, however, metadata management, table consistency, schema evolution, partition changes, and historical versions become important engineering concerns.
Apache Iceberg addresses these challenges by introducing a structured table format based on:
Table Metadata
↓
Snapshots
↓
Manifest Lists
↓
Manifest Files
↓
Data Files
Enter fullscreen mode Exit fullscreen mode
The three ideas I would remember are:
- Iceberg separates the logical table from its physical file organization.
- Snapshots and manifests provide structured table state and enable capabilities such as time travel.
- Schema evolution, hidden partitioning, and partition evolution make large analytical tables easier to manage as workloads change.
In short:
Better metadata + reliable table state + scalable evolution = a stronger foundation for the modern data lakehouse.