24 - Caching - MemoryStore Redis

Welcome to Day 24 of Learn GCP in 30 Days! Over the last two days, you mastered persistent storage with relational SQL (Cloud SQL) and NoSQL documents (Firestore). Today, we supercharge application performance with In-Memory Caching using Google Cloud Memorystore (Redis).

🛑 The Core Problem: The 50ms Database Bottleneck

Every time a user visits your e-commerce homepage or opens an article, your backend server executes database queries (e.g., SELECT * FROM products WHERE featured = true;).

What Existed Previously:

Applications queried persistent databases on every single HTTP request. Even if 100,000 users viewed the exact same product banner or news headline within 10 seconds, the database executed the exact same expensive disk read 100,000 times.

Problems Faced:

• 🐢 Disk I/O Latency: Persistent SSDs and hard drives take 10ms to 50ms to locate data blocks, parse SQL syntax, and return results.
• 💥 Database Meltdown During Spikes: Flash sales or breaking news create thousands of queries per second (QPS), exhausting database connection pools and maxing out CPU.
• 💸 Expensive Vertical Scaling: To survive traffic spikes, engineering teams were forced to upgrade databases to massive, costly 64-core VMs.
• 📉 Slow User Experience: High latency causes web pages to load sluggishly, hurting conversion rates and user retention.

How Present Technology Solves It:

Google created Cloud Memorystore (Fully Managed In-Memory Data Store):

1. Blazing Sub-Millisecond Speed: RAM is over 1,000x faster than SSD disks. Memorystore serves read/write queries in < 1 millisecond ($0.5\text{ms}$).
2. Offload 90% of Database Traffic: By caching popular database query results in RAM, your underlying database only handles 5% to 10% of total read traffic.
3. Zero Maintenance Overhead: Google handles node provisioning, replication, automated failover, and OS security patching.
4. Rich In-Memory Capabilities: Beyond simple key-value caching, Redis provides native data structures for real-time gaming leaderboards, pub/sub messaging, user session storage, and API rate limiting.

🗄️ Real-World Analogy: The Desk vs. The Basement Archive

• Cloud SQL / Firestore = The Basement Archive:
  • Secure, permanent, fireproof storage that holds millions of files. But walking to the basement for every single customer question is exhausting and slow.
• Cloud Memorystore Redis = The Sticky Notes on Your Desk:
  • You keep the 5 most frequently needed phone numbers and pricing sheets right on your desk.
  • When a customer asks a common question, you answer in half a second without leaving your chair!

⚡ Redis vs. Memcached: Which Should You Choose?

Google Cloud Memorystore supports the two industry-standard open-source caching engines:

🔄 The #1 Architecture: The Cache-Aside Pattern (Lazy Loading)

The Cache-Aside Pattern is the most widely used caching strategy in software engineering.

Follow the Data:

1. Step 1 (Check Cache): The application checks Redis for the key product:101.
2. Step 2A (Cache Hit - ~95% of requests): Data exists in Redis. The app returns it to the user in 0.5 milliseconds. The SQL database is never touched!
3. Step 2B (Cache Miss - ~5% of requests): Data is not in Redis (e.g., first time requested or cache expired).
4. Step 3 (Fetch & Populate): The app queries Cloud SQL, retrieves the record, and immediately writes it into Redis with an expiration time (TTL - Time-To-Live, e.g., 300 seconds).
5. Future Requests: The next 10,000 users reading product:101 get instant Cache Hits from RAM!

🏗️ Core Architecture: Service Tiers & VPC Isolation

1. Memorystore Service Tiers

• Basic Tier:
  • Single standalone Redis node in one availability zone.
  • Ideal for development, testing, and non-critical cache workloads (lowest cost).
• Standard Tier (High Availability):
  • Provisions a Primary Node and a Read Replica across two distinct zones.
  • Continuous asynchronous replication with automated failover in $< 30$ seconds and a 99.9% SLA.

2. Network Isolation (Zero Public IP)

🧪 Hands-on Lab: Provisioning Memorystore Redis & Python Caching

In this lab, you will create a lightweight 1 GB Memorystore Redis instance, connect to it from a Compute VM in the same VPC, and execute Python code demonstrating the Cache-Aside Pattern and Redis Leaderboards.

Step 1: Provision a Lightweight 1 GB Redis Instance

Pathway A: Web Console (Click-by-Click)

1. Open the Google Cloud Console (https://console.cloud.google.com/).
2. In the top search bar, type Memorystore and select Memorystore for Redis (or navigate to Navigation Menu $\rightarrow$ Databases $\rightarrow$ Memorystore $\rightarrow$ Redis).
3. Click CREATE INSTANCE.
4. Configure Instance Settings:
   • Instance ID: gcp-day24-redis
   • Tier: Select Basic (lowest cost for development).
   • Capacity: Enter 1 GB.
   • Region: Choose us-central1 (Iowa) (or your preferred region).
   • Network: Select default.
5. Click CREATE. (Provisioning takes about 2 to 3 minutes).

Pathway B: Cloud Shell CLI

Open Cloud Shell and run:

# 1. Set environment variables
export PROJECT_ID=$(gcloud config get-value project)
export INSTANCE_NAME="gcp-day24-redis"
export REGION="us-central1"

# 2. Create a 1 GB Basic tier Redis instance
gcloud redis instances create ${INSTANCE_NAME} \
  --size=1 \
  --region=${REGION} \
  --tier=basic \
  --network=default

Step 2: Create a Test Compute VM to Access Redis

Because Memorystore has no public IP, we will spin up a small, temporary test VM in the same default VPC to interact with Redis.

# Create a tiny e2-micro test VM
gcloud compute instances create redis-test-vm \
  --zone=${REGION}-a \
  --machine-type=e2-micro \
  --subnet=default

# Get the Private IP address of your Redis instance
export REDIS_IP=$(gcloud redis instances describe ${INSTANCE_NAME} --region=${REGION} --format="value(host)")
echo "👉 Your Redis Private IP is: ${REDIS_IP}"

Step 3: SSH into VM & Write the Cache-Aside Python Script

SSH into your test VM:

gcloud compute ssh redis-test-vm --zone=${REGION}-a

Inside the VM terminal, let's install Python Redis tools:

# 1. Install Redis CLI and Python library
sudo apt update && sudo apt install -y redis-tools python3-pip
pip3 install redis --break-system-packages --quiet

export REDIS_HOST="10.x.x.x"

# 2. Create the Cache-Aside Simulation script
cat << 'EOF' > cache_demo.py
import time
import os
import redis

# Connect to Google Cloud Memorystore Redis
redis_host = os.getenv("REDIS_HOST", "127.0.0.1")
cache = redis.Redis(host=redis_host, port=6379, decode_responses=True)

def simulate_slow_database_query(product_id):
    """Simulates a heavy 50ms disk SQL query."""
    time.sleep(0.05)  # 50ms simulated disk latency
    return f"Premium Wireless Headphones (SKU-{product_id}) - $199.99"

def get_product_details(product_id):
    cache_key = f"product:{product_id}"
    
    # Step 1: Check Redis Cache
    start_time = time.time()
    cached_data = cache.get(cache_key)
    
    if cached_data:
        elapsed = (time.time() - start_time) * 1000
        print(f"  ⚡ [CACHE HIT] {cached_data} | Time: {elapsed:.2f} ms")
        return cached_data
    
    # Step 2: Cache Miss - Query Database
    print(f"  🐢 [CACHE MISS] Querying disk database for {cache_key}...")
    db_data = simulate_slow_database_query(product_id)
    
    # Step 3: Store in Redis with a 60-second TTL (Time-To-Live)
    cache.setex(cache_key, 60, db_data)
    
    elapsed = (time.time() - start_time) * 1000
    print(f"  💾 Data saved to Redis. Total Time: {elapsed:.2f} ms")
    return db_data

print("==================================================")
print("🚀 Simulating 1st Request (Empty Cache):")
get_product_details(881)

print("\n🚀 Simulating 2nd Request (Cached in RAM):")
get_product_details(881)

print("\n🚀 Simulating 3rd Request (Cached in RAM):")
get_product_details(881)
print("==================================================")
EOF

# 4. Run the demo script
python3 cache_demo.py

Expected Output:

==================================================
🚀 Simulating 1st Request (Empty Cache):
  🐢 [CACHE MISS] Querying disk database for product:881...
  💾 Data saved to Redis. Total Time: 52.14 ms

🚀 Simulating 2nd Request (Cached in RAM):
  ⚡ [CACHE HIT] Premium Wireless Headphones (SKU-881) - $199.99 | Time: 0.62 ms

🚀 Simulating 3rd Request (Cached in RAM):
  ⚡ [CACHE HIT] Premium Wireless Headphones (SKU-881) - $199.99 | Time: 0.48 ms
==================================================

(Notice how latency dropped from 52 ms down to 0.48 ms—a 100x performance increase!)

Step 4: Real-Time Gaming Leaderboard with Redis Sorted Sets

Redis has built-in support for Sorted Sets (ZADD), making it the worldwide industry standard for gaming leaderboards and real-time rankings:

Run the following command to create and run the leaderboard script:

cat << 'EOF' > leaderboard.py
import os
import redis

redis_host = os.getenv("REDIS_HOST", "127.0.0.1")
cache = redis.Redis(host=redis_host, port=6379, decode_responses=True)

# Add player scores to a Sorted Set named 'game_leaderboard'
cache.zadd("game_leaderboard", {
    "Player_Alex": 4500,
    "Player_Mani": 9800,
    "Player_Sarah": 7200,
    "Player_Bruce": 8900
})

print("🏆 Top 3 Players on Leaderboard:")
# Fetch top 3 players with highest scores in descending order
top_players = cache.zrevrange("game_leaderboard", 0, 2, withscores=True)

for rank, (player, score) in enumerate(top_players, start=1):
    print(f"  #{rank}: {player} — {int(score)} pts")
EOF

python3 leaderboard.py

Expected Output:

🏆 Top 3 Players on Leaderboard:
  #1: Player_Mani — 9800 pts
  #2: Player_Bruce — 8900 pts
  #3: Player_Sarah — 7200 pts

Exit the VM:

exit

🛡️ Step 5: Credit Safety & Resource Teardown

Pathway A: Web Console (Click-by-Click)

1. Go to Memorystore $\rightarrow$ Redis.
2. Click the checkbox next to gcp-day24-redis.
3. Click DELETE at the top toolbar $\rightarrow$ Confirm deletion.
4. Go to Compute Engine $\rightarrow$ VM Instances $\rightarrow$ Delete redis-test-vm.

Pathway B: Cloud Shell CLI

# 1. Delete the Memorystore Redis instance
gcloud redis instances delete ${INSTANCE_NAME} --region=${REGION} --quiet

# 2. Delete the temporary test VM
gcloud compute instances delete redis-test-vm --zone=${REGION}-a --quiet

📝 Day 24 Summary & Quick Reference

Core Architecture Takeaways

1. In-Memory Speed: RAM provides sub-millisecond (< 1ms) read/write speeds, eliminating disk I/O bottlenecks.
2. Cache-Aside Pattern: Applications check Redis first (Cache Hit). On a Cache Miss, they fetch from SQL/NoSQL, write to Redis with a TTL, and return.
3. Redis vs. Memcached: Redis is the preferred choice for 95% of modern apps due to rich data structures (Hashes, Sorted Sets, Pub/Sub) and High Availability.
4. VPC Isolation: Memorystore has zero public IPs and is secured inside your private VPC network.
5. Cost Optimization: Always tear down test instances to preserve $300 trial credits.

Quick Command Cheat Sheet

Tomorrow, in Day 25, we explore globally distributed, massive-scale enterprise databases with Google Cloud Spanner (Global Relational SQL) and Cloud Bigtable (Petabyte NoSQL for Time-Series & IoT)!

← 23 - NoSQL DBs - Firestore for Realtime Apps | Next Topic → 25 - Enterprise DBs - Spanner and Bigtable Overview