Lab 14: Infrastructure as Code with Terraform

Before we write code, let's understand our Goal: Imagine you are hired to build a massive Databricks platform for a bank. You click around the UI, create the Workspace, set up Unity Catalog, and add 500 users. Tomorrow, your boss says, "Great! Now do the exact same thing for the European branch." Do you want to manually click buttons for another 5 days? What if you click the wrong button?

The Tool: Terraform is an Infrastructure-as-Code (IaC) tool. It allows you to write down exactly what infrastructure you want, and it automatically builds it in the cloud.

1. Infrastructure as Code (IaC)

Real-World Analogy Mapping: Imagine building a Skyscraper.

• The Long Way (ClickOps): You tell a construction worker, "Put a brick here, now put a window there, now paint the wall blue." If the building falls down, you have no idea what went wrong because there was no plan.
• The Smart Way (Terraform): You hand the construction worker a Blueprint (A .tf text file). The worker looks at the blueprint and builds the entire skyscraper automatically. If you want a second skyscraper, you just hand them a copy of the blueprint.

2. The Databricks Terraform Blueprint

In Terraform, we write configuration files (usually called main.tf) using HashiCorp Configuration Language (HCL).

Anatomy Breakdown: Here is the blueprint for creating a Unity Catalog Metastore, a Production Catalog, and a Finance Schema.

# 1. The Provider: Tell Terraform we are connecting to Databricks
terraform {
  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = "1.28.0"
    }
  }
}

# 2. The Foundation: Create the Metastore
resource "databricks_metastore" "enterprise_metastore" {
  name          = "primary-metastore"
  storage_root  = "s3://enterprise-metastore-bucket/metastore"
  region        = "us-east-1"
}

# 3. The Floor: Create the Production Catalog
resource "databricks_catalog" "prod_catalog" {
  metastore_id = databricks_metastore.enterprise_metastore.id
  name         = "prod_data"
  comment      = "Production data managed by Terraform blueprint"
}

# 4. The Room: Create a Schema inside the Catalog
resource "databricks_schema" "finance_schema" {
  catalog_name = databricks_catalog.prod_catalog.name
  name         = "finance"
}

3. The Execution Commands

Once you have written your blueprint, how do you actually tell the "construction workers" to start building?

A Platform Engineer runs these three commands in their terminal:

1. terraform init : Downloads the Databricks tools (Hires the workers).
2. terraform plan : Reads your blueprint and prints out exactly what it is going to build or destroy, allowing you to double-check the math (Reviews the blueprint).
3. terraform apply : Actually connects to the Databricks API and builds the infrastructure.

Magic Feature: If a junior employee accidentally deletes the prod_data catalog in the Databricks UI, the Platform Engineer just runs terraform apply again. Terraform will notice the catalog is missing from reality, look at the blueprint, and instantly rebuild it!

← Previous: Lab 13: Generative AI and Vector Search | Next: Lab 15: Edge Cases & Concurrency Control →**