20 - Hands-on Lab - Serverless API Pipeline

Welcome to Day 20 of Learn GCP in 30 Days! Today is your Week 3 Capstone Hands-on Lab.

🏗️ The Problem: The Blocking Synchronous E-Commerce Checkout

Throughout Week 3, we explored how modern cloud applications handle unpredictable traffic spikes without crashing servers or bankrupting companies.

What Existed Previously:

Traditional web applications were built as monolithic, synchronous servers. When a user clicked "Place Order" or submitted a payment:

Problems Faced:

• ⏳ High Latency & Freezing: The user had to stare at a spinning loading spinner for 5 to 10 seconds while the server completed every background task sequentially.
• 💥 Cascading Failures: If the third-party SMS notification provider was slow or down, the entire checkout request timed out and the customer was told their purchase failed!
• 💸 Wasted Idle Costs: High-capacity virtual machines were kept running 24/7 at 100% cost just in case a traffic surge arrived.
• 🐌 Poor Scalability: When thousands of buyers flooded a flash sale, the monolithic web servers ran out of memory and crashed.

How Present Technology Solves It:

By building a Decoupled Serverless Event-Driven Pipeline on Google Cloud:

1. Ingestion API on Cloud Run: A lightweight, containerized Python web service receives incoming order requests, validates them, publishes an event to a Pub/Sub topic in 5 milliseconds, and immediately returns 200 OK (Order Confirmed) to the user.
2. Pub/Sub Shock Absorber: Buffers millions of incoming messages safely with zero data loss, absorbing traffic spikes effortlessly.
3. Eventarc Event Router: Automatically listens for new messages on the topic and routes them as standard CloudEvents to backend workers.
4. Asynchronous Cloud Run Function (2nd Gen): Wakes up on demand, reads the event payload, executes background processing (inventory update, confirmation logging), and scales down to zero ($0.00) when idle.
5. Artifact Registry: Provides a private, secure Docker repository holding our application container images.

🍕 Real-World Analogy: The Fast-Food Drive-Through & Kitchen Display

Think of this serverless pipeline like a modern High-Speed Drive-Through Restaurant:

Real-World Analogy Mapping:

• 🎙️ The Drive-Through Cashier = Cloud Run Ingestion API: The cashier takes your order, taps a button on the touch terminal, and immediately hands you your receipt with your order number. The cashier does not ask you to sit at the intercom speaker for 10 minutes while they personally cook your burger.
• 📺 The Kitchen Ticket Display Screen = Pub/Sub & Eventarc: As soon as the order is tapped into the system, an electronic ticket pops up on the kitchen screen. If 50 cars arrive at once, tickets queue up neatly on the screen without getting lost.
• 👨‍🍳 The Kitchen Chefs = Cloud Run Functions: The cooks standing at the grill see the ticket appear, prepare the food, and package the meal. When there are no cars in line, the kitchen staff rests (zero cost). When 50 orders arrive, extra cooks jump in to prepare meals concurrently (auto-scaling)!

🚆 Architectural Deep-Dive: The Railway Ticketing System

A common question is: "What if we are building a train or airline reservation system? Must the user wait for a seat to be allocated before receiving a response?"

The short answer: Yes! For inventory-critical systems, you divide the workflow into two distinct phases:

1. Phase 1: What Must Wait (Synchronous Seat Lock ~50ms)

• The passenger clicks "Book Seat".
• Cloud Run holds the connection for 50 milliseconds while executing an ACID transaction on the database: SELECT seat FROM coaches WHERE seat = 'B2-14' AND status = 'AVAILABLE' FOR UPDATE;.
• Once the database confirms the seat is temporarily locked, Cloud Run responds immediately with 200 OK or 202 Accepted ({"status": "SEAT_LOCKED", "pnr": "PNR-49201", "seat": "B2-14"}).
• The user's screen updates immediately with their confirmed seat number.

2. Phase 2: What Must Never Wait (Asynchronous Fulfillment 3-5 seconds)

Now that the seat is safely reserved, the user does not stay on hold for slow downstream tasks:

• PDF & QR Code Generation (1,500ms) $\rightarrow$ Cloud Function Worker 1.
• WhatsApp & SMS Dispatch (2,000ms) $\rightarrow$ Cloud Function Worker 2.
• Railway Chart & Analytics Update (800ms) $\rightarrow$ Cloud Function Worker 3.

🚄 The Ultra-Peak "Tatkal" Dilemma: Queue-First Model

When 500,000 users try to book only 200 seats at 10:00 AM on the dot, hitting the database directly with 500,000 simultaneous locks will crash the database.

In this extreme situation:

1. Cloud Run does not touch the database. It immediately dumps booking requests into Pub/Sub (FIFO Queue) in order of millisecond arrival and returns: "HTTP 202: You are in queue at position #42. Waiting for confirmation...".
2. A single high-speed backend worker reads from the queue sequentially, books seats one-by-one with zero database collisions, and notifies each user via WebSockets when their turn succeeds.

🗺️ Capstone Lab Architecture: What We Are Building

In this lab, you will build and test a 6-step serverless pipeline:

🧪 Step-by-Step Hands-on Lab Activity

You can complete this capstone lab using either the Web Console UI (Click-by-Click) or the Cloud Shell CLI (Fast Track)!

Step 1: Enable the Required Serverless APIs

Google Cloud requires specific API services to be enabled before creating container registries, Cloud Run services, Pub/Sub topics, and Eventarc triggers.

Option A: Web Console UI

1. Open console.cloud.google.com.
2. Ensure your correct lab project is selected at the top.
3. In the top search bar, search for APIs & Services and click Enabled APIs & services.
4. Click + ENABLE APIS AND SERVICES and enable each of the following:
   • Artifact Registry API (artifactregistry.googleapis.com)
   • Cloud Run Admin API (run.googleapis.com)
   • Cloud Functions API (cloudfunctions.googleapis.com)
   • Cloud Pub/Sub API (pubsub.googleapis.com)
   • Eventarc API (eventarc.googleapis.com)
   • Cloud Build API (cloudbuild.googleapis.com)

Option B: Cloud Shell CLI (Fast Track)

Open Cloud Shell (click the >_ terminal icon in the top-right corner) and execute:

gcloud services enable \
  artifactregistry.googleapis.com \
  run.googleapis.com \
  cloudfunctions.googleapis.com \
  pubsub.googleapis.com \
  eventarc.googleapis.com \
  cloudbuild.googleapis.com

Anatomy Breakdown:

Step 2: Create Artifact Registry Repository & Prepare Code

We will create a Docker repository named serverless-repo to store our container images.

Option A: Web Console UI

1. Search for Artifact Registry in the top search bar and click it.
2. Click + CREATE REPOSITORY.
3. Fill in the repository configuration:
   • Name: serverless-repo
   • Format: Docker
   • Mode: Standard
   • Location type: Region
   • Region: us-central1 (or your preferred region)
4. Click CREATE.

Option B: Cloud Shell CLI

Run the following command in Cloud Shell:

gcloud artifacts repositories create serverless-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Docker repository for Serverless Capstone Lab"

Step 3: Create the Pub/Sub Messaging Topic

Now, let's create the central shock absorber topic where all incoming order events will land.

Option A: Web Console UI

1. Search for Pub/Sub in the top search bar and select Topics.
2. Click + CREATE TOPIC.
3. Set Topic ID to order-events-topic.
4. Leave all default encryption and schema settings unchanged.
5. Click CREATE.

Option B: Cloud Shell CLI

Run the following command in Cloud Shell:

gcloud pubsub topics create order-events-topic

Step 4: Build & Deploy the Ingestion API on Cloud Run

Next, we create the frontend microservice that exposes a public REST API endpoint (POST /orders). When an order arrives, it publishes a JSON message to order-events-topic and returns an immediate response.

Let's create the microservice files inside Cloud Shell:

mkdir -p ~/serverless-lab/api && cd ~/serverless-lab/api

Create the application code file app.py (nano app.py or Graphical Editor):

import os
import json
from flask import Flask, request, jsonify
from google.cloud import pubsub_v1

app = Flask(__name__)

# Retrieve environment variables
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
TOPIC_ID = os.environ.get("TOPIC_ID", "order-events-topic")

# Initialize Pub/Sub Publisher Client
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(PROJECT_ID, TOPIC_ID)

@app.route("/", methods=["GET"])
def health_check():
    return jsonify({
        "status": "healthy",
        "service": "Order Ingestion API",
        "mode": "Serverless Cloud Run"
    }), 200

@app.route("/orders", methods=["POST"])
def place_order():
    request_json = request.get_json(silent=True)
    if not request_json:
        return jsonify({"error": "Invalid request body. JSON required."}), 400

    order_id = request_json.get("order_id", "ORD-UNKNOWN")
    item = request_json.get("item", "Generic Item")
    amount = request_json.get("amount", 0.0)

    # Prepare event payload
    event_payload = {
        "order_id": order_id,
        "item": item,
        "amount": amount,
        "status": "RECEIVED",
        "source": "CloudRun-Ingestion-API"
    }

    # Publish message to Pub/Sub asynchronously (Shock Absorber)
    message_data = json.dumps(event_payload).encode("utf-8")
    future = publisher.publish(topic_path, message_data)
    message_id = future.result()

    # Immediate non-blocking response to customer
    return jsonify({
        "message": "Order successfully accepted into serverless pipeline!",
        "order_id": order_id,
        "pubsub_message_id": message_id,
        "processing_mode": "Asynchronous Eventarc Pipeline"
    }), 202

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 8080))
    app.run(host="0.0.0.0", port=port)

Anatomy Breakdown of app.py:

Create requirements.txt (nano requirements.txt $\rightarrow$ paste $\rightarrow$ Ctrl+O $\rightarrow$ Enter $\rightarrow$ Ctrl+X):

Flask==3.0.3
google-cloud-pubsub==2.21.1
gunicorn==22.0.0

Create Dockerfile (nano Dockerfile $\rightarrow$ paste $\rightarrow$ Ctrl+O $\rightarrow$ Enter $\rightarrow$ Ctrl+X):

FROM python:3.11-slim

ENV PYTHONUNBUFFERED=True
WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 app:app

Anatomy Breakdown of Dockerfile:

Build & Deploy the API Service

Run the following build and deployment commands in Cloud Shell:

# 1. Build and push container to Artifact Registry using Cloud Build
PROJECT_ID=$(gcloud config get-value project)

gcloud builds submit --tag us-central1-docker.pkg.dev/$PROJECT_ID/serverless-repo/order-api:v1

# 2. Deploy the container to Cloud Run
gcloud run deploy order-ingestion-api \
  --image us-central1-docker.pkg.dev/$PROJECT_ID/serverless-repo/order-api:v1 \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars GOOGLE_CLOUD_PROJECT=$PROJECT_ID,TOPIC_ID=order-events-topic

Step 5: Deploy the Event-Driven Cloud Run Function (2nd Gen)

Now, we create the downstream backend worker using Cloud Run Functions (2nd Gen). This function will be triggered automatically whenever a new message is published to order-events-topic.

Let's set up the function workspace in Cloud Shell:

mkdir -p ~/serverless-lab/worker && cd ~/serverless-lab/worker

Create the function entry file main.py (nano main.py $\rightarrow$ paste $\rightarrow$ Ctrl+O $\rightarrow$ Enter $\rightarrow$ Ctrl+X):

import base64
import json
import functions_framework

# Triggered from a message on a Cloud Pub/Sub topic via Eventarc
@functions_framework.cloud_event
def process_order_event(cloud_event):
    # Extract Pub/Sub message data from the CloudEvent
    pubsub_data = cloud_event.data.get("message", {}).get("data", "")
    
    if pubsub_data:
        # Decode base64 binary payload to UTF-8 JSON
        decoded_string = base64.b64decode(pubsub_data).decode("utf-8")
        event_json = json.loads(decoded_string)
        
        order_id = event_json.get("order_id", "UNKNOWN")
        item = event_json.get("item", "UNKNOWN")
        amount = event_json.get("amount", 0.0)
        source = event_json.get("source", "UNKNOWN")
        
        # Simulate background business logic (e.g., inventory, fraud detection, database write)
        print(f"==================================================")
        print(f"🎉 [EVENTARC WORKER] Processing Order Event!")
        print(f"📦 Order ID       : {order_id}")
        print(f"🛒 Purchased Item : {item}")
        print(f"💰 Amount         : ${amount}")
        print(f"📡 Event Source   : {source}")
        print(f"✅ Order fulfillment completed successfully!")
        print(f"==================================================")
    else:
        print("⚠️ Warning: Received empty Pub/Sub message data.")

Anatomy Breakdown of main.py:

Create requirements.txt for the worker:

functions-framework==3.8.1

Deploy the Function with Pub/Sub Trigger

Run the deployment command in Cloud Shell:

gcloud functions deploy order-processor-worker \
  --gen2 \
  --runtime python311 \
  --region us-central1 \
  --source . \
  --entry-point process_order_event \
  --trigger-topic order-events-topic

Step 6: Test & Verify the End-to-End Pipeline

Let's test our complete event-driven pipeline by firing live orders into our Cloud Run API and watching the downstream Cloud Function process them asynchronously.

1. Retrieve the Cloud Run API URL

Run the following command to get your public API URL:

API_URL=$(gcloud run services describe order-ingestion-api --region us-central1 --format='value(status.url)')
echo "Your Live API URL is: $API_URL"

2. Test the API Health Check Endpoint

curl -i $API_URL/

You will see an immediate 200 OK JSON response:

{
  "mode": "Serverless Cloud Run",
  "service": "Order Ingestion API",
  "status": "healthy"
}

3. Send Multiple Asynchronous Orders

Let's post 3 simulated customer orders:

# Order 1
curl -X POST $API_URL/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-9001", "item": "Google Pixel 9 Pro", "amount": 999.00}'

echo ""

# Order 2
curl -X POST $API_URL/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-9002", "item": "Chromebook Plus", "amount": 499.50}'

echo ""

# Order 3
curl -X POST $API_URL/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-9003", "item": "Pixel Buds Pro 2", "amount": 229.00}'

Notice how every request returns in under 50 milliseconds with an HTTP 202 Accepted and a unique pubsub_message_id:

{
  "message": "Order successfully accepted into serverless pipeline!",
  "order_id": "ORD-9001",
  "processing_mode": "Asynchronous Eventarc Pipeline",
  "pubsub_message_id": "11823908129038"
}

4. Verify Real-Time Worker Execution in Cloud Logging

Now, let's check the logs of our background Cloud Function worker to confirm that Eventarc routed the messages and the function executed successfully:

gcloud functions logs read order-processor-worker \
  --gen2 \
  --region us-central1 \
  --limit 30

You will see output resembling:

==================================================
🎉 [EVENTARC WORKER] Processing Order Event!
📦 Order ID       : ORD-9001
🛒 Purchased Item : Google Pixel 9 Pro
💰 Amount         : $999.0
📡 Event Source   : CloudRun-Ingestion-API
✅ Order fulfillment completed successfully!
==================================================

5. Verify in Web Console UI

1. Open the GCP Console.
2. Navigate to Cloud Run $\rightarrow$ Click on order-ingestion-api $\rightarrow$ View the Metrics tab to observe incoming request traffic graphs.
3. Navigate to Cloud Functions $\rightarrow$ Click on order-processor-worker $\rightarrow$ Click the Logs tab to view the live colored terminal output of your processed events!

🧹 Step 7: Credit Safety Teardown & Cleanup Steps ($0.00 Guarantee)

To guarantee that your $300 Free Trial credits remain 100% safe and zero ongoing costs are incurred, delete the resources created during this lab.

Option A: Web Console UI

1. Cloud Functions: Go to Cloud Functions $\rightarrow$ Select order-processor-worker $\rightarrow$ Click DELETE.
2. Cloud Run: Go to Cloud Run $\rightarrow$ Select order-ingestion-api $\rightarrow$ Click DELETE.
3. Pub/Sub: Go to Pub/Sub $\rightarrow$ Topics $\rightarrow$ Select order-events-topic $\rightarrow$ Click DELETE.
4. Artifact Registry: Go to Artifact Registry $\rightarrow$ Select serverless-repo $\rightarrow$ Click DELETE.

Option B: Cloud Shell CLI (Fast Track Cleanup)

Run these commands in Cloud Shell:

# 1. Delete Cloud Function (2nd Gen)
gcloud functions delete order-processor-worker --gen2 --region=us-central1 --quiet

# 2. Delete Cloud Run API Service
gcloud run services delete order-ingestion-api --region=us-central1 --quiet

# 3. Delete Pub/Sub Topic (and auto-generated subscriptions)
gcloud pubsub topics delete order-events-topic --quiet

# 4. Delete Artifact Registry Repository and container images
gcloud artifacts repositories delete serverless-repo --location=us-central1 --quiet

# 5. Clean up local lab directories
rm -rf ~/serverless-lab

Signal vs. Noise: Key Concepts & Noise Filter

Common Doubts & Interview Traps

Q1: Why did we use Cloud Run for the Ingestion API and Cloud Functions for the worker instead of putting everything into a single service?

• Answer: Single-responsibility microservice design! The Ingestion API is optimized for high-concurrency public HTTP traffic with ultra-low latency. The worker is optimized for event-driven background processing that can be updated, scaled, or replaced without touching the public API.

Q2: What is the main advantage of Pub/Sub acting as a shock absorber during a flash sale?

• Answer: If 50,000 users place orders simultaneously, the Ingestion API accepts all 50,000 requests in seconds without crashing. Pub/Sub safely buffers the messages, allowing backend workers to process them steadily without overwhelming downstream databases or external APIs.

Q3: What is the difference between Cloud Functions (2nd Gen) and Cloud Run?

• Answer: Under the hood, Cloud Functions (2nd Gen) actually runs on top of Cloud Run! Cloud Functions offers a simplified developer experience (you just write a Python/Node.js function and Google builds the container), while Cloud Run gives you full control over custom Docker containers, multiple listening ports, and complex web server binaries.

Q4: What happens if the background Cloud Function worker fails or throws an exception while processing a message?

• Answer: By default, if an event-triggered function fails, Pub/Sub will retry delivering the message after an acknowledgment deadline. You can configure a Dead Letter Queue (DLQ) in Pub/Sub to catch and inspect problematic messages after a specified number of failed attempts.

Daily Practice Drill & Self-Check

Test your understanding of today's capstone lab:

// Try answering these:
1. Which HTTP status code is best practice when an API accepts a request and enqueues it for asynchronous background processing?
2. What open standard event format is used by Eventarc to route events between GCP services?
3. If an application receives zero requests for 4 hours, what is the compute cost of the Cloud Run API and Cloud Function worker?

🎉 Huge Congratulations! You have completed Day 20 and graduated from Week 3: Serverless & Containerized Applications!

Over the past 6 days, you have mastered:

• Docker Container fundamentals & Artifact Registry (Day 15)
• Serverless Container deployment on Cloud Run (Day 16)
• Event-Driven microservices with Cloud Functions (Day 17)
• Asynchronous messaging with Pub/Sub & Eventarc (Day 18)
• Kubernetes architecture & GKE Autopilot (Day 19)
• Built a complete, production-grade Serverless Event-Driven API Pipeline (Day 20)!

Take some time to celebrate your achievements. Tomorrow, we kick off Week 4: Storage & Relational/NoSQL Databases starting with Day 21: Storage 101 - Cloud Storage (GCS) Buckets!

← 19 - GKE Kubernetes Overview for Beginners | Next Topic → 21 - Storage 101 - Cloud Storage (GCS) Buckets