Back to all blogs
    4 min read

    🔐 Data Governance (Unity Catalog)

    #databricks#governance#unity-catalog#security

    Data Governance (Unity Catalog)

    🏛️ The Evolution of Governance

    What Existed Previously: Organizations had independent workspaces for different teams, and each workspace had its own isolated local Hive Metastore.

    Problems Faced: Security teams had to manually duplicate and maintain access controls across dozens of workspaces. If an employee left the company, their access had to be manually revoked in every individual workspace, leading to severe compliance risks and fragmented data silos.

    How Present Technology Solves It: Databricks introduces Unity Catalog (UC), a centralized governance hub that eliminates siloed governance. It manages the entire Data Lake, Metadata, Data Warehouse, Machine Learning Assets, and AI Assets under one unified security model used by every single workspace and persona in the company. It integrates seamlessly with external Identity Providers (Azure Entra ID, Okta) via SCIM Sync.

    🔬 Anatomy Breakdown: The Metastore Hierarchy

    Unity Catalog introduces a strict, multi-level namespace hierarchy to organize enterprise assets:

    1. Metastore: The top-level container for metadata.
    2. Catalog: The first level of data isolation (e.g., prod_catalog, dev_catalog).
    3. Schema (Database): The second level of isolation (e.g., finance_schema).
    4. Tables / Views / Volumes: The actual data objects.
      • Tables/Views: Standard tabular data.
      • Volumes: A specialized UC object designed to securely govern non-tabular, unstructured files in object storage (e.g., raw JSON, PDFs, images).

    Managed vs. External Tables:

    • Managed Tables: Databricks controls both the metadata and the actual storage path.
      • Crucial Detail: If you run DROP TABLE on a Managed Table in Unity Catalog, it deletes the metadata AND completely destroys the underlying data files.
    • External Tables: Databricks only manages the metadata. If you drop the table, the underlying files remain safely in your cloud storage (S3/ADLS).

    🕸️ Data Lineage & Delta Sharing

    Automated Data Lineage: Unity Catalog automatically tracks data movement across ETL jobs, SQL queries, notebooks, and Delta Live Tables. It tracks lineage down to the exact Column Level.

    • Impact Analysis (Looking Forward): If a Data Engineer wants to change a column in a Bronze table, Lineage shows exactly which downstream executive dashboards will break.
    • Root Cause Analysis (Looking Backward): If a CEO notices an error in a Gold dashboard, engineers can trace the error backwards through the Silver layer directly to the source system.

    Delta Sharing (B2B Data Exchange): Delta Sharing securely exposes datasets internally and externally (B2B, partners, customers) without ever copying or replicating the data.

    • How it works: Consumers request a table, and the Delta Sharing Server returns Pre-Signed Temporary URLs. The consumer then reads the data directly from the original cloud storage bucket as Parquet files.
    • Use Case: Breaking internal silos (Finance securely sharing a live table with Sales) or Data Monetization (selling access to data products).

    🧪 Practice Drill

    Q1. A Data Engineer executes the DROP TABLE command on a Unity Catalog Managed Table. What happens to the underlying Parquet files in cloud storage?

    Q2. You want to securely grant a Data Scientist access to a folder containing 5,000 raw MRI images (unstructured data) using standard SQL GRANT commands. Which specific Unity Catalog object should you use?

    Q3. Your company wants to sell live market data to external clients, but you refuse to constantly copy and FTP files to them. What Databricks feature solves this?

    💡 Click for Solutions

    A1. They are permanently deleted. Dropping a Managed Table deletes both the metadata and the actual underlying data files. (If it were an External Table, only the metadata would be deleted).

    A2. A Volume. Volumes are specialized Unity Catalog objects designed specifically to govern unstructured, non-tabular files.

    A3. Delta Sharing. It allows external clients to securely query your live data directly without you ever having to copy, replicate, or move the data.

    ← 🚰 Data Engineering & Pipelines | Next Topic → 🛡️ The Security Layer