21 - Storage 101 - Cloud Storage (GCS) Buckets

Welcome to Day 21 of Learn GCP in 30 Days! Today, we kick off Week 4: Storage & Relational/NoSQL Databases. Over the next 6 days, you will master how data is stored, queried, and protected in Google Cloud.

We begin with the most foundational storage service in Google Cloud: Google Cloud Storage (GCS).

🛑 The Core Problem: Hard Drives Don't Scale to Petabytes

When building applications, you constantly need to store unstructured files: profile pictures, uploaded PDFs, video recordings, analytics logs, and daily database backups.

What Existed Previously:

Traditional servers store data on attached hard drives (Block Storage) or shared network folders (File Storage / NFS).

Problems Faced:

• 🛑 Fixed Size Limits: A hard disk has a strict maximum capacity (e.g., 500 GB). When your users upload 501 GB of images, the disk fills up and your application crashes.
• 💸 Over-Provisioning Waste: To prevent disk full errors, teams pre-purchased expensive 5 TB disks for systems that only used 50 GB.
• 🔒 No Direct Web Access: Files on a hard drive cannot be directly accessed over the internet via HTTP/HTTPS without running a dedicated web server (like NGINX or Apache) in front of them.
• 💥 Single Point of Failure: If a physical hard drive fails or a sector corrupts, data can be permanently lost unless manual RAID arrays or off-site backups are maintained.

How Present Technology Solves It:

Google created Cloud Storage (GCS) - Object Storage:

1. Infinite Elasticity: You never choose a disk size. Store 1 byte or 100 petabytes; GCS scales automatically with zero capacity planning.
2. Global Web Accessibility: Every file uploaded receives a unique HTTPS URL (e.g., https://storage.googleapis.com/your-bucket/photo.png).
3. Extreme 11 9's Durability: GCS offers 99.999999999% annual durability by automatically replicating data across multiple physical devices and availability zones.
4. Pay-As-You-Go: You only pay for the exact gigabytes stored per month (starting at ~$0.02 per GB/month for active data and under$0.0012 per GB/month for archive data).

🏢 Real-World Analogy: The 3 Types of Cloud Storage

To truly understand Cloud Storage, let's compare the three fundamental storage types in cloud computing:

🎟️ The Coat Check Analogy for Object Storage

Imagine checking a coat at a hotel banquet:

1. You hand over your coat (the Data Payload).
2. The attendant tags it with your name, phone number, and ticket number (the Metadata).
3. You receive a claim ticket #8942 (the Unique Key / URL).
4. When you return the ticket #8942, the attendant retrieves your exact coat instantly. It doesn't matter where on the 5,000 coat racks it was placed; the key resolves directly to the object.

🪣 Anatomy of Google Cloud Storage

1. Buckets

A Bucket is the root container that holds your files.

• Globally Unique Name: Bucket names share a global namespace across all Google Cloud customers worldwide (like website domain names). Once my-cool-bucket is taken, no one else in the world can create a bucket with that name until it is deleted.
• Naming Constraints: Must contain 3 to 63 lowercase letters, numbers, hyphens, and underscores. No uppercase letters or spaces allowed.
• Location Type:
  • Region (e.g., us-central1): Lowest cost, lowest latency for compute resources in that specific region.
  • Dual-Region (e.g., nam4 - Iowa & South Carolina): High availability and automatic failover across two geographic regions.
  • Multi-Region (e.g., US or EU): Highest availability; geo-redundantly distributed across an entire continent.

2. Objects (Files)

An Object is a piece of data stored in a bucket. An object consists of:

• Data (Payload): The actual bytes of the file (text, image, MP4 video, zip archive). Maximum size for a single object is 5 Terabytes.
• Key (Name): The unique string identifier (e.g., users/avatar/profile_101.jpg).
• Metadata: Key-value pairs describing the object (e.g., Content-Type: image/jpeg, Cache-Control, custom user labels).

3. Flat Namespace (No Real Folders!)

4. Object Immutability

Objects in GCS are immutable (cannot be modified in-place).

• You cannot append 10 lines to an existing 100 MB text file in GCS.
• To update an object, you upload a complete new version of the file, which atomically replaces the old object.

💰 The 4 GCS Storage Classes: Tiering for Extreme Savings

Not all data has the same access frequency. A logo loaded 10,000 times a minute on your homepage should not be priced the same as an IRS financial audit backup accessed once every 5 years.

Google provides 4 Storage Classes to optimize your cloud bill:

🤖 Google Cloud Autoclass (Smart Auto-Tiering)

Don't want to manually guess when files become cold?

• Autoclass: An automated toggle on a bucket that monitors each object's access patterns.
• If an object isn't accessed for 30 days, GCS automatically transitions it to Nearline.
• If untouched for 90 days, it moves to Coldline; at 365 days, to Archive.
• When an application suddenly reads an archived file, Autoclass transitions it back to Standard with zero retrieval fees!

🔄 Object Lifecycle Management: Automate Your Data Pipeline

Object Lifecycle Management allows you to set declarative rules on your bucket so GCS manages data age automatically.

Anatomy of a Lifecycle Rule

Every rule combines an Action with one or more Conditions:

1. Actions:

   • SetStorageClass: Change the storage tier (e.g., transition Standard $\rightarrow$ Nearline or Archive).
   • Delete: Permanently purge the object or delete non-current versions.
   • AbortIncompleteMultipartUpload: Cancel and delete incomplete multi-part file uploads older than $X$ days to prevent orphan data charges.

2. Conditions:

   • Age: Number of days since the object was created (e.g., age: 30).
   • CreatedBefore: A specific calendar date (e.g., files created before 2025-01-01).
   • MatchesPrefix / MatchesSuffix: Apply rule only to specific paths (e.g., temp/ or .log).
   • NumberOfNewerVersions: Used with Object Versioning (e.g., keep only the latest 3 versions).

🔐 Security & Access Control: Protecting Your Data

1. Uniform Bucket-Level Access (Google Recommended)

• In the past, GCS used complex Access Control Lists (ACLs) per object, leading to security misconfigurations.
• Uniform Bucket-Level Access unifies all security under GCP IAM. You grant roles (like Storage Object Viewer or Storage Object Admin) at the bucket level or project level. Individual files inherit bucket permissions.

2. Public vs. Private Buckets

• By default, all GCS buckets and objects are 100% private. Only authorized IAM identities can access them.
• To make a bucket public for hosting static website assets, you grant the special identity allUsers the role roles/storage.objectViewer.

3. Signed URLs (Temporary Access Without IAM)

What if a customer needs to download a private invoice PDF or upload their profile photo, but they don't have a Google Cloud account?

• Signed URL: A cryptographically signed HTTPS link that grants temporary read or write access for a specific duration (e.g., 15 minutes).
• Once the time expires, the URL becomes invalid immediately.

⚡ The Modern CLI: gcloud storage vs. Legacy gsutil

What Existed Previously:

For over a decade, developers used the standalone Python CLI tool gsutil (e.g., gsutil cp file.txt gs://bucket/).

Problems Faced:

gsutil required a separate Python runtime, was slower for multi-gigabyte transfers, and had a command syntax distinct from the rest of gcloud.

How Present Technology Solves It:

Google introduced gcloud storage, a modern, high-performance CLI integrated directly into the gcloud SDK. It provides up to 94% faster parallel upload and download speeds and uniform command syntax.

The Long Way (Legacy gsutil):

# Legacy syntax (slower, older tool)
gsutil mb -l us-central1 gs://my-legacy-bucket-991
gsutil cp my-image.png gs://my-legacy-bucket-991/
gsutil ls gs://my-legacy-bucket-991/

The Smart Way (Modern gcloud storage):

# Modern gcloud storage syntax (lightning fast, native gcloud)
gcloud storage buckets create gs://my-modern-bucket-991 --location=us-central1
gcloud storage cp my-image.png gs://my-modern-bucket-991/
gcloud storage objects list gs://my-modern-bucket-991/

🧪 Hands-on Lab: Provisioning Buckets, Uploads, and Lifecycle Automation

In this hands-on lab, you will create a Cloud Storage bucket, upload and inspect objects, configure automated lifecycle rules, and test security controls.

Step 1: Create a Globally Unique Bucket

Pathway A: Web Console (Click-by-Click)

1. Open the Google Cloud Console (https://console.cloud.google.com/).
2. In the top search bar, type Cloud Storage and select Buckets (or navigate to Navigation Menu $\rightarrow$ Cloud Storage $\rightarrow$ Buckets).
3. Click + CREATE at the top.
4. Configure your bucket settings:
   • Name your bucket: gcp-day21-[YOUR_NAME]-[RANDOM_NUMBERS] (e.g., gcp-day21-mani-88219). Click Continue.
   • Choose where to store your data:
     • Location type: Select Region.
     • Location: Select us-central1 (Iowa) (or your preferred region). Click Continue.
   • Choose a storage class for your data:
     • Select Set a default class $\rightarrow$ choose Standard. Click Continue.
   • Choose how to control access to objects:
     • Check Enforce public access prevention on this bucket.
     • Access control: Select Uniform (Recommended). Click Continue.
   • Choose how to protect object data:
     • Object versioning: Leave None (default).
     • Encryption: Leave Google-managed encryption key (default).
5. Click CREATE.

Pathway B: Cloud Shell CLI

Open Google Cloud Shell and run:

# Set your environment variables
export PROJECT_ID=$(gcloud config get-value project)
export BUCKET_NAME="gcp-day21-lab-${PROJECT_ID}"
export REGION="us-central1"

# Create the bucket using modern gcloud storage CLI
gcloud storage buckets create gs://${BUCKET_NAME} \
  --location=${REGION} \
  --default-storage-class=STANDARD \
  --uniform-bucket-level-access

Step 2: Create a Local Test File and Upload Objects

Let's create a sample text file and upload it to your new bucket.

In Cloud Shell:

# 1. Create a sample text file
cat << 'EOF' > welcome.txt
==================================================
  Welcome to Day 21 of Learn GCP in 30 Days!
  Service: Google Cloud Storage (GCS)
  Durability: 99.999999999% (11 9's)
  Storage Class: Standard Object Storage
==================================================
EOF

# 2. Upload the file to your GCS bucket
gcloud storage cp welcome.txt gs://${BUCKET_NAME}/

# 3. Create a simulated folder by uploading with a prefix
gcloud storage cp welcome.txt gs://${BUCKET_NAME}/documents/day21-notes.txt

# 4. List all objects in the bucket
gcloud storage objects list gs://${BUCKET_NAME}/

Pathway A: Web Console (Upload Verification)

1. In the Cloud Console, click into your newly created bucket name.
2. Under the Objects tab, you will see welcome.txt and a simulated folder icon named documents/.
3. Click on welcome.txt to view its metadata: Size, Content-Type (text/plain), Storage Class (Standard), and Creation Time.
4. Click the AUTHENTICATED URL link to open and view the file contents in your browser!

Step 3: Configure Object Lifecycle Management

Let's automate cost savings by adding a Lifecycle Rule:

• Automatically downgrade objects to Nearline after 30 days.
• Automatically delete old temporary objects after 365 days.

Pathway A: Web Console (Click-by-Click)

1. In your bucket details page, click on the LIFECYCLE tab.
2. Click + ADD A RULE.
3. Under Select an action:
   • Select Set storage class to Nearline. Click Continue.
4. Under Select conditions:
   • Check Age. Enter 30 (days). Click Continue.
5. Click CREATE RULE.
6. (Optional) Click + ADD A RULE again, select Delete object, set Age to 365 days, and click CREATE RULE.

Pathway B: Cloud Shell CLI

We can define lifecycle rules declaratively via a JSON configuration file.

Run the following command to create the lifecycle rule configuration:

cat << 'EOF' > lifecycle.json
{
  "rule": [
    {
      "action": {
        "type": "SetStorageClass",
        "storageClass": "NEARLINE"
      },
      "condition": {
        "age": 30,
        "matchesStorageClass": ["STANDARD"]
      }
    },
    {
      "action": {
        "type": "Delete"
      },
      "condition": {
        "age": 365
      }
    }
  ]
}
EOF

Apply the lifecycle JSON to your bucket:

# Apply the lifecycle configuration
gcloud storage buckets update gs://${BUCKET_NAME} --lifecycle-file=lifecycle.json

# Describe bucket to verify the lifecycle rules are active
gcloud storage buckets describe gs://${BUCKET_NAME} --format="json(lifecycle_config)"

Step 4: Test Object Metadata and Storage Class Inspection

Let's inspect how GCS displays object metadata and update an object's storage class manually:

# 1. View detailed metadata of the uploaded object
gcloud storage objects describe gs://${BUCKET_NAME}/welcome.txt

# 2. Manually change an individual object's storage class to ARCHIVE
gcloud storage objects update gs://${BUCKET_NAME}/welcome.txt --storage-class=ARCHIVE

# 3. Verify the updated storage class
gcloud storage objects describe gs://${BUCKET_NAME}/welcome.txt --format="value(storage_class)"

Output:

ARCHIVE

🛡️ Step 5: Credit Safety & Resource Teardown

To ensure **$0.00** ongoing costs against your$300 Free Trial credits, delete the bucket and its objects when you have finished experimenting.

Pathway A: Web Console (Click-by-Click)

1. Navigate to Cloud Storage $\rightarrow$ Buckets.
2. Locate your bucket in the list (gcp-day21-...).
3. Click the checkbox next to the bucket name.
4. Click the DELETE button at the top toolbar.
5. In the confirmation dialog, type DELETE and confirm.

Pathway B: Cloud Shell CLI

# Delete all objects and the bucket in one command
gcloud storage rm --recursive gs://${BUCKET_NAME}

# Clean up local scratch files
rm -f welcome.txt lifecycle.json

# Verify bucket is completely removed
gcloud storage buckets list

📝 Day 21 Summary & Quick Reference

Core Architecture Takeaways

1. Object Storage vs. Block/File: GCS is a flat namespace designed for unstructured data (images, videos, backups) with unlimited elasticity and 11 9's durability.
2. Globally Unique Names: Bucket names are shared across the entire global Google Cloud ecosystem.
3. 4 Storage Classes: Standard (hot), Nearline (30d warm), Coldline (90d cold), and Archive (365d frozen).
4. Lifecycle Management: Declarative JSON rules automate tiering and deletion without writing background cron jobs.
5. Modern CLI: Always prefer gcloud storage over legacy gsutil for faster, multi-threaded operations.

Quick Command Cheat Sheet

Tomorrow, in Day 22, we dive into relational databases with Cloud SQL (PostgreSQL & MySQL)—learning how to provision managed database engines, handle automated backups, and connect backend applications!

← 20 - Hands-on Lab - Serverless API Pipeline | Next Topic → 22 - Relational DBs - Cloud SQL (Postgres and MySQL)