Back to all blogs
    2 min read

    Lab 09: Testing & Monitoring

    #databricks#testing#monitoring#pytest

    Before we write code, let's understand our Goal: We need to mathematically prove our code works before it touches production data, and we need to monitor who is deleting tables or spending too much money.

    The Tools:

    • Pytest: A standard Python framework for writing Unit Tests.
    • System Tables: Built-in hidden tables in Databricks that record every action, audit log, and billing event.

    1. PySpark Unit Testing

    Real-World Analogy Mapping: Imagine a Car Factory. The Long Way: Building the entire car, turning the key, and hoping the engine doesn't explode. If it explodes, you have to tear the whole car apart to find the bad spark plug. The Smart Way (Unit Testing): Testing the spark plug on a test bench before putting it in the car.

    In PySpark, we isolate a single function, feed it fake data, and assert that the output matches our expectations.

    python
    # The Function we want to test
def remove_nulls(df):
    return df.dropna(subset=["id"])

# The Unit Test (usually saved in a file called test_data_prep.py)
def test_remove_nulls(spark):
    # 1. Arrange (Create the fake mock data)
    mock_data = [(1, "Alice"), (None, "Bob")]
    df = spark.createDataFrame(mock_data, ["id", "name"])
    
    # 2. Act (Run our function)
    cleaned_df = remove_nulls(df)
    
    # 3. Assert (Prove the math is correct)
    # We expect 'Bob' to be removed because his ID is None.
    assert cleaned_df.count() == 1
    assert cleaned_df.collect()[0]["name"] == "Alice"

    Engineers run these tests automatically in their CI/CD pipelines. If a test fails, the deployment is blocked!

    2. System Tables (Observability)

    Problems Faced: In the past, if a junior engineer accidentally deleted a production table, it was nearly impossible to figure out who did it. Furthermore, if the CFO asked "Why is our cloud bill so high this month?", engineers had to guess.

    How Present Technology Solves It: Unity Catalog automatically populates System Tables. These tables track every single event across the entire Databricks platform. You can query them using standard SQL!

    Cost Monitoring

    Query the billing table to see exactly which workspace and cluster is burning the most Databricks Units (DBUs).

    sql
    -- Query compute cost usage for the last 30 days
SELECT workspace_id, sku_name, SUM(usage_quantity) as total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY workspace_id, sku_name;

    Audit Logs (The Security Camera)

    Query the audit log to find out who executed a destructive command.

    sql
    -- Find out who deleted a table
SELECT event_time, user_identity.email, action_name 
FROM system.access.audit
WHERE action_name = 'delete_table';

    ← Previous: Lab 08: Orchestration, CI/CD & DABs | Next: Lab 10: MLflow & MLOps Expansion →**