🏗️ Data Architectures: Data Lakes, Mesh & Medallion

🏗️ Data Architectures

Different architectures exist because different businesses have different needs — size, speed, cost, flexibility.

1️⃣ Enterprise Data Warehouse (EDW)

A single, centralized warehouse that serves the entire organization — all departments, all subjects, all data in one place.

All Sources → ETL → One Central EDW → All Departments query it

Characteristics:

• Organization-wide scope (not just one department)
• Single source of truth for the entire enterprise
• Governed, structured, historical data
• Typically built on: Snowflake, Teradata, IBM Db2, Oracle

Best for:

• Large enterprises needing consistent data across all departments
• Heavily regulated industries (Banking, Insurance, Healthcare)

2️⃣ Data Marts

A department-scoped subset of the EDW (covered in Ch04).

EDW → [subset] → Sales Mart
    → [subset] → Finance Mart
    → [subset] → HR Mart

3️⃣ Data Lake

A Data Lake stores raw, unprocessed data in its native format — structured, semi-structured, and unstructured — at massive scale.

Best for:

• Storing massive volumes of diverse data cheaply
• ML/AI workloads that need raw data
• When you don't know exactly what questions you'll ask yet

4️⃣ Lakehouse

A Lakehouse combines the best of Data Lake and Data Warehouse into one architecture.

Technologies:

• Delta Lake (Databricks)
• Apache Iceberg
• Apache Hudi

5️⃣ Data Mesh

Data Mesh is a decentralized architecture where data ownership is distributed to individual domain teams (e.g., Sales team owns Sales data, HR team owns HR data).

4 Core Principles:

1. Domain ownership — each team owns, builds, and maintains their own data products
2. Data as a product — each domain treats its data as a product with SLAs and documentation
3. Self-serve platform — central team provides infrastructure; domains use it themselves
4. Federated governance — global standards enforced, but each domain has autonomy

Sales Domain  → owns → Sales Data Product
Finance Domain → owns → Finance Data Product
HR Domain     → owns → HR Data Product
         ↓ shared via common standards
     Consumers query across domains

Best for:

• Large organizations with many autonomous teams
• When a central data team becomes a bottleneck

6️⃣ Medallion Architecture (Bronze → Silver → Gold)

Medallion Architecture is a layered data design pattern used inside Lakehouses (especially Databricks Delta Lake) that progressively refines data quality across 3 layers.

Source Data
    ↓
🥉 Bronze  — raw, as-is (JSON logs, CSV dumps, API responses)
    ↓ clean + deduplicate
🥈 Silver  — validated, standardized, joined tables
    ↓ aggregate + model
🥇 Gold    — business-ready fact/dimension tables, KPIs, dashboards

Real Example (E-commerce):

Bronze → Raw order events from Kafka (JSON, duplicates, nulls)
Silver → Cleaned orders table (deduped, currency standardized, nulls handled)
Gold   → Daily_Sales_Summary table (SUM revenue by region/product — feeds Power BI)

7️⃣ Virtual Data Warehouse

A Virtual Data Warehouse does not physically store data — instead it creates a virtual layer on top of existing source systems and presents them as if they were a warehouse.

Source DB 1 ─┐
Source DB 2 ─┤→ Virtual Layer (query federation) → Analyst queries
Source DB 3 ─┘

Best for:

• Quick implementation with no ETL pipelines
• When data cannot be moved (regulatory, privacy reasons)
• Ad hoc analysis on existing systems

⚔️ DW vs EDW

🗺️ Architecture Comparison Summary

🧪 Practice Drill

// Try answering these:
// Q1. What is the key difference between a Data Lake and a Data Warehouse in terms of schema?
// Q2. A startup stores raw clickstream logs, images, and CSV sales data all in one place cheaply for future ML use. Which architecture are they using?
// Q3. Name the 3 layers of Medallion Architecture and what each contains.
// Q4. A large bank has 20 departments all querying inconsistent data from their own databases. Which architecture solves this at scale?
// Q5. Why would a company choose a Virtual Data Warehouse? What is its main limitation?
// Q6. Match the architecture to its best use case:
// | Use Case | Architecture |
// |---|---|
// | Each domain team owns and serves its own data | ? |
// | Progressive data refinement: raw → clean → curated | ? |
// | One warehouse for the entire global enterprise | ? |
// | BI + ML on the same platform without data movement | ? |

← Previous Topic | Next Topic → Next Topic