Modern Data Architectures (Delta Lake & Iceberg)

🌊 The Messy Data Lake

What Existed Previously: Companies used Data Lakes (like HDFS or Amazon S3) to dump massive amounts of raw, unstructured, and structured data cheaply.

Analogy

Imagine a physical lake where you throw in everything: clean water, old boots, and rusty bicycles. It's easy to throw things in, but incredibly hard to find clean water when you need a drink.

Problems Faced:

No ACID Transactions: If a data pipeline failed halfway through writing, the Data Lake was left with corrupted, partial data. There was no way to safely "rollback".
No Schema Enforcement: Anyone could write a file where the "Age" column was text instead of a number, breaking all the downstream reporting dashboards.
No "Update" or "Delete": Data Lakes are fundamentally just folders of files. If a user requested their data be deleted (e.g., for GDPR compliance), you couldn't just run a DELETE SQL command. You had to manually rewrite entire massive files.

How Present Technology Solves It: Lakehouse Architectures (like Delta Lake and Apache Iceberg) were created to add the reliability and structure of a traditional database on top of the cheap storage of a Data Lake.

🏗️ Delta Lake

What is it? Delta Lake is an open-source storage layer that sits on top of your existing Data Lake. It physically saves your data as fast Parquet files, but keeps a transaction log alongside them tracking every single change made.

✨ Key Features of Delta Lake

1️⃣ ACID Transactions

If a write job fails at 99%, Delta Lake reads the transaction log and rolls back the entire transaction. You never get partial, corrupted data.

2️⃣ Schema Enforcement & Evolution

Enforcement: Delta Lake refuses to write data if the new columns don't perfectly match the existing table structure.
Evolution: If you intentionally want to add a new column, you can safely evolve the schema without breaking old data.

3️⃣ Time Travel

Because Delta Lake tracks every change in a log, you can easily query what the data looked like yesterday, or completely undo accidental deletions!

python

# Practical Example: Time Travel in Delta Lake

# Read data as it exists right now
df_current = spark.read.format("delta").load("/path/to/table")

# Read what the data looked like exactly at version 0 (before accidental deletes!)
df_history = spark.read.format("delta").option("versionAsOf", 0).load("/path/to/table")

🧊 Apache Iceberg

What Existed Previously: As companies got even larger, Delta Lake and Hive had limitations when scaling to millions of files and petabytes of data, primarily because tracking files in massive directories became a bottleneck.

Problems Faced: Querying a table with millions of partitions took a very long time just to figure out which files to read before the actual query even started executing.

How Present Technology Solves It: Apache Iceberg is another Lakehouse format (like Delta Lake) but built explicitly for massive, petabyte-scale tables.

Analogy

If Delta Lake is a highly organized filing cabinet, Iceberg is an automated, robotic warehouse where the index is so smart that it instantly knows exactly which box out of millions contains your file without ever having to look inside the folders.

🌟 Why choose Iceberg?

Hidden Partitioning: Users don't need to know how the data is partitioned (e.g., by day or month) to write fast queries. Iceberg handles the routing silently.
Incredible Scale: It doesn't rely on directory structures to find files; it uses a tree of highly optimized metadata files, making query planning for massive datasets nearly instant.

🥇 The Medallion Architecture

Follow the Data: As data moves through a modern Lakehouse, it should be progressively cleaned and refined.

Bronze Layer (Raw): The dumping ground. Data is ingested exactly as it arrives (JSON, CSV) with no cleaning. If the pipeline breaks, you can always replay from Bronze.
Silver Layer (Cleansed): Data is filtered, deduplicated, and converted into Delta/Iceberg format. It is now queryable for basic analysis.
Gold Layer (Curated): Highly refined business-level tables. Aggregated for specific BI dashboards (e.g., daily_sales_by_region).

🤖 Advanced Pipeline Automation

1️⃣ Event-Driven Webhooks

What Existed Previously: Running a script on a schedule (e.g., every night at 2 AM) using cron jobs. Problems Faced: Data is delayed. If the file arrives at 2:05 AM, it won't be processed until the next day. How Present Technology Solves It: Webhooks. When a file lands in the Data Lake, it instantly triggers a webhook event that automatically starts the Spark pipeline.

2️⃣ GitLab CI/CD for Iceberg

The Goal: You want to automatically update Iceberg table schemas in production without manually running SQL commands. The Solution: You define your Iceberg schema in a YAML file. When a developer updates the YAML and pushes it to GitLab, a CI/CD pipeline automatically authenticates via IAM roles and safely updates the Iceberg table in production.

🕰️ Slowly Changing Dimensions (SCD)

When data changes over time (e.g., a customer moves to a new city), how do you store the change in your Lakehouse?

Type	How it works	When to use it
SCD Type 1 (Overwrite)	Simply overwrites the old record. The old city is completely erased and replaced by the new city.	When you only care about the current state and don't care about history.
SCD Type 2 (Versioning)	Adds a new row with the new city. Both the old and new rows exist, but have `start_date` and `end_date` columns to track exactly when they lived where.	When you must track historical changes for auditing or accurate historical reporting.

🧪 Practice Drill

text

// Try answering these:
**Q1.** A developer accidentally deletes a million rows in a Data Lake. Why is this a disaster in a traditional Data Lake, but an easy fix in Delta Lake?

**Q2.** What feature of Delta Lake prevents a user from writing a `String` into an `Integer` column?

**Q3.** Both Delta Lake and Iceberg try to bring traditional database features to the Data Lake. What are those traditional features collectively called? (Hint: It's an acronym).

💡 Click for Solutions

A1. A traditional Data Lake has no rollback mechanism; the files are simply gone or corrupted. Delta Lake has Time Travel, allowing you to read or restore the data exactly as it was in a previous version.

A2. Schema Enforcement.

A3. ACID Transactions (Atomicity, Consistency, Isolation, Durability).

← Advanced SQL & Transformations | Syllabus → Syllabus