Login

  • Yesterday

Open Table Format — Iceberg, Delta, Hudi

Delta Lake, Apache Iceberg, and Apache Hudi have emerged to bridge the gap between the flexibility and scalability of data lakes and the reliability and strong data management features of traditional data warehouses.

What is Open Table format?

An open table format is an open-source specification that sits between your physical data files (like Parquet, ORC, or Avro) and query engines (like Spark, Trino, Snowflake, or DuckDB). It manages table metadata — tracking files, schema evolution, and row updates — so object storage (AWS S3, Azure ADLS, Google Cloud Storage) can behave like an ACID-compliant relational database.

And why are they useful?

Before open table formats, companies tried to turn cheap object storage (like AWS S3) into massive databases using traditional frameworks (like Apache Hive). But treating a simple folder of files like a database caused massive headaches.

To understand why treating a simple folder of files like a database caused massive headaches, let’s look at a concrete real-world example: an e-commerce orders table.

Imagine you manage an online store’s data lake. Every purchase, order update, and cancellation gets stored in cloud storage (s3://my-company-data/orders/) as standard Parquet files, partitioned by date.

Under the traditional setup (the Apache Hive pattern), the query engine assumes “Every file inside s3://.../orders/ is a valid part of the table."

Here is how that assumption breaks down in production across four real scenarios.

1. The “Half-Finished Write” Nightmare (No Atomic Commits)

The Scenario:

At midnight, an automated ETL job processes 10,000 new orders for May 2nd and attempts to write 10 new Parquet files into date=2024-05-02/.

Halfway through the job — after writing 5 of the 10 files — the ETL node crashes or loses network connectivity.

The Headache:

  • Dirty Reads: If an executive runs an automated morning dashboard report, it will read those 5 files and calculate 50% of the actual revenue for the day.

  • Messy Cleanup: S3 has no concept of a “rollback.” The failed 5 files sit in the directory forever, corrupting future queries unless an engineer manually goes in, identifies which files are orphaned, and deletes them.

2. The GDPR “Delete Request” Disaster (No Row-Level Operations)

The Scenario:

A user named Alice exercises her legal right under GDPR/CCPA to have her account deleted. Her order history is scattered across 500 Parquet files over the last 3 years.

The Traditional Approach:

Because Parquet files are immutable (you cannot modify existing text inside a Parquet file), you cannot simply run DELETE FROM orders WHERE user_id = 'Alice'.

Instead, your script must:

  1. Scan all 500 Parquet files containing her records.

  2. Read every single row into memory.

  3. Filter out Alice’s rows.

  4. Write 500 brand-new Parquet files back to S3.

  5. Delete the 500 old Parquet files.

The Headache:

To remove 10 rows of data, you just read and rewrote hundreds of gigabytes of data. Doing this daily for hundreds of deletion requests generates massive cloud compute bills and disk I/O strain.

3. The Column Rename Incident (No Metadata Layer)

The Scenario:

An engineer decides to standardize the schema. They want to rename the column customer_email to user_email.

In a database like Postgres or Snowflake, this takes 1 second (ALTER TABLE orders RENAME COLUMN...).

In a folder-based data lake:

  • The engineer updates the schema in the Hive Metastore to user_email.

  • New data pipeline runs start writing files with the user_email header.

The Headache:

Older Parquet files still physically contain the internal header customer_email. When a analyst runs a query spanning both historical and new partitions:

The query engine reads the older files, fails to find the column user_email, and returns NULL for every historical record (or crashes entirely).

To fix it, you would have to rewrite every historical Parquet file in the entire storage bucket to update the file metadata header.

4. The “Listing S3” Scalability Wall (No File Pruning)

The Scenario:

Your e-commerce platform grows over 5 years. You now have 100,000 Parquet files organized into deep subdirectories (year/month/day/hour/).

An analyst wants to answer a simple question: “How many orders were placed between 2:00 PM and 3:00 PM today?”

The Traditional Approach:

To figure out which files to read, the query engine must issue hundreds of LIST API requests to S3 to recursively inspect the folder tree.

The Headache:

S3 rate-limits LIST requests. Listing 100,000 files can take 3 to 5 minutes and cost money in S3 API calls before the query engine reads a single byte of actual data.

How Open Table Formats Fix This (The Metadata Solution)

Open table formats solve every one of these headaches by placing a single, authoritative metadata file over the folder.

Traditional Hive metastores relied on directory listing calls (LIST operations in object storage) to discover files for a query—an operation that scales poorly and introduces consistency latency.

Open table formats eliminate LIST operations entirely by using hierarchical metadata files

  • Atomic Commits: File C is ignored by readers until the Metadata file updates to include it. If a job fails halfway, File C is simply left out of the metadata log — zero corrupt reads.

  • Row Updates: Small “delete vectors” or delta files mark Alice’s specific row as deleted in metadata without requiring a total rewrite of 100GB files.

  • Schema Mapping: Column renames map the old ID to the new name in the metadata layer — no physical file rewrites needed.

  • No Directory Listing: The query engine reads one manifest file to know the exact locations and statistics of all data files instantly.

What is the high level Architecture?

Below is how Open table formats accomplishes using hierarchical metadata files:

  • Catalog / Snapshot Pointer: Points to the latest valid metadata file (representing the state at a specific commit).

  • Metadata File: Stores table properties, current schema, partition specs, and pointers to manifest lists.

  • Manifest List: Enumerates the individual manifest files that form a snapshot, tracking key metrics (e.g., min/max column values per file) for aggressive partition and file pruning.

  • Data Files: The underlying physical files storing the raw records (typically standard Apache Parquet or ORC).

Core Distinction and Key Technical Differences

  • Traditional File Formats (Directory-Based): Define how individual files store records. A “table” is simply a folder/directory in a file system or object store containing those files.

  • Open Table Formats (Metadata-Driven): Define how files interact as a unified, stateful table. A “table” is defined by an explicit manifest/metadata tree, regardless of where physical files reside.

Comparison — Open Table format and Traditional file format

But, where to use which open table format?

Now that we have established that there is no escaping our Good old days ACID property, let’s deep dive on the 3 popular open table format we have.

To understand where to use which open table format, it helps to look at their founding design principles. While feature parity has grown across all three (and tools like Apache XTable and Delta UniForm make metadata translation easier), each format retains a distinct core strength:

  • Apache Iceberg: Built for multi-engine flexibility & strict schema evolution.

  • Delta Lake: Built for Databricks, Spark ecosystems & SQL warehouse performance.

  • Apache Hudi: Built for high-frequency CDC, continuous streaming & record-level upserts.

1. Quick Summary: Which One to Use Where?

2. Practical Example: The Rideshare & Food Delivery Platform

To see how these distinctions play out in practice, imagine an app like Uber or DoorDash with three distinct data pipelines processing driver locations, order updates, and analytics:

                                  +-----------------------+
                                  |   EVENT DATA SOURCE   |
                                  +-----------+-----------+
                                              |
      +---------------------------------------+---------------------------------------+
      |                                       |                                       |
      v                                       v                                       v
[ PIPELINE A: CDC UPSERTS ]        [ PIPELINE B: BATCH & SPARK ]           [ PIPELINE C: MULTI-ENGINE ]
High-frequency record updates      Databricks ML & ETL Jobs                 Trino, Snowflake & Flink
"Update order status to 'Delivered'" "Train driver dispatch model"          "Ad-hoc BI dashboards & queries"
      |                                       |                                       |
      v                                       v                                       v
+---------------------------+     +---------------------------+     +---------------------------+
|        APACHE HUDI        |     |        DELTA LAKE         |     |      APACHE ICEBERG       |
| Record-level index avoids |     | Native to Databricks;     |     | Single table queryable by |
| full file rewrites.       |     | tight Spark integration.  |     | Snowflake, Trino & Python |
+---------------------------+     +---------------------------+     +---------------------------+

Scenario A: High-Frequency CDC (Choose Apache Hudi)

Requirement: 50,000 drivers send status updates every minute (“Searching”, “En Route”, “Delivered”). You receive a raw database CDC stream from Postgres via Debezium/Kafka.

Why Hudi wins:

  • Record-Level Indexing: Hudi maintains a key-value index that points directly to the exact physical file containing driver_id = 12345. It doesn't need to search partitions.

  • Merge-on-Read (MoR): Instead of rewriting a 100MB Parquet file every time a driver changes status, Hudi appends a tiny delta log. In the background, it asynchronously merges these log files into standard Parquet without blocking incoming streams.

Scenario B: Unified Spark & Databricks Platform (Choose Delta Lake)

Requirement: A team of Data Scientists and Data Engineers build feature engineering pipelines and machine learning dispatch models using Apache Spark and Databricks.

Why Delta Lake wins:

  • Engine Optimization: Delta Lake was engineered specifically around Apache Spark’s internal APIs. Writes, merges, and queries execute with minimal overhead.

  • Turnkey Managed Services: On platforms like Databricks or Microsoft Fabric, compaction, table optimization (OPTIMIZE ZORDER / Liquid Clustering), and transaction management run automatically in the background without needing manual tuning.

Scenario C: Multi-Engine Enterprise Analytics (Choose Apache Iceberg)

Requirement: The central data warehouse serves multiple distinct tools:

  • Flink processes streaming orders.

  • Spark runs nightly batch transformations.

  • Snowflake and Trino run ad-hoc business intelligence queries.

  • Data Scientists run local analytical scripts in Python using DuckDB/PyIceberg.

Why Iceberg wins:

  • Engine Agnostic & Open Governance: Iceberg was designed from day one as a neutral specification managed by the Apache Software Foundation. It treats all engines as first-class citizens.

  • Hidden Partitioning & Partition Evolution: If you originally partitioned tables by day (year-month-day) and later decide to re-partition by hour (year-month-day-hour) because volume doubled, Iceberg updates the partition layout as a metadata-only change. Queries filtering by date do not need to change their WHERE clause syntax, and old data doesn't need to be rewritten.

3. Core Technical Differences at a Glance

          ICEBERG                         DELTA LAKE                     HUDI
          +---------------------+         +---------------------+        +---------------------+
Primary   | Engine Agnostic &   |         | Databricks / Spark  |        | High-Velocity CDC & |
Focus     | Multi-Tool Standard |         | Performance & Simplicity     | Streaming Ingestion |
          +----------+----------+         +----------+----------+        +----------+----------+
                     |                               |                              |
Metadata  | Hierarchical Tree   |         | Append-only Log     |        | Timeline & Action   |
Structure | (Catalog -> Meta -> |         | (JSON logs +        |        | Instants (.hoodie)  |
          |  Manifests) [1.1.4] |         |  Parquet DVs) [1.1.4|        | + Key Index [1.1.4] |
                     |                               |                              |
Updates   | Copy-on-Write or    |         | Copy-on-Write or    |        | First-class MoR     |
Strategy  | Deletion Vectors    |         | Deletion Vectors    |        | (Base Parquet +     |
          | (V3) [1.1.4, 1.1.6] |         | (DVs) [1.1.6]       |        |  Delta Logs) [1.1.4]|

4. Architectural Summary

  1. Pick Iceberg if you are building a modern, cloud-native data lakehouse using multiple tools (e.g., Snowflake + AWS Athena + Spark + DuckDB) and want complete vendor neutrality and schema/partition evolution guarantees.

  2. Pick Delta Lake if your data team operates predominantly inside Apache Spark, Databricks, or Microsoft Fabric, and wants the smoothest setup with minimal maintenance.

  3. Pick Hudi if your primary challenge is ingestion latency — streaming continuous CDC data from operational databases with heavy row-level updates.