Planning and configuring data storage options

Study guide for Associate Cloud Engineer, Unit 2 · Topic 2. This is the topic's lecture in reading form — every slide's teaching, figures and worked examples, in order — followed by the official Google Cloud pages its claims rest on.

What the exam guide asks, quoted. Planning and configuring data storage options. Considerations include:

1. Product choice (e.g., Cloud SQL, BigQuery, Firestore, Spanner, Bigtable)
2. Choosing storage options (e.g., zonal Persistent Disk, regional Persistent Disk, Standard, Nearline, Coldline, Archive)

Planning and configuring data storage options

Two build-time choices: which product, and which storage option

Which product holds the data — Cloud SQL, Spanner, Firestore, Bigtable or BigQuery, read from the shape of the data and how it is used. Which storage option holds the bytes — a zonal or regional Persistent Disk under a virtual machine, or a Cloud Storage class chosen by how often the data is read.

The Associate Cloud Engineer guide frames this topic as planning and configuring data storage options, and it lists only two considerations. The first is product choice, with five examples: Cloud SQL, BigQuery, Firestore, Spanner and Bigtable. The second is choosing storage options, with six examples: zonal Persistent Disk, regional Persistent Disk, and the four Cloud Storage classes Standard, Nearline, Coldline and Archive. The Cloud Digital Leader exam asks what these products are for; this exam asks you to provision the right one. So the deck teaches the signals in a scenario that pick a product, then the flags and defaults that decide what you actually get when you create a disk or a bucket.

Five products, five shapes of data

Read the data's shape and its use before you pick the product

Worked example (synthetic). An engineer is handed three requests in one week: move a PostgreSQL order system, store readings from a fleet of meters, and let analysts query five years of sales. That is Cloud SQL, Bigtable and BigQuery — three products, because three shapes of data.

Start with what each product is, because the exam gives you a scenario and expects you to hear the product in it. Cloud SQL is a fully managed relational database service for MySQL, PostgreSQL and SQL Server — it handles backups, high availability and failover, and maintenance and updates for you — and Google's advice is to use it if you want to spend less time managing your database and more time using it. Spanner is also relational, but it brings together relational, graph, key-value and search, and it offers transactional consistency at global scale with automatic, synchronous replication. Firestore is a fully managed document database that does not require any pre-provisioning of resources and auto scales to match your load. Bigtable is ideal for storing large amounts of single-keyed data with low latency, with high read and write throughput. And BigQuery combines a cloud-based data warehouse and powerful analytic tools, storing data in a columnar format optimized for analytical queries. Two of these are relational, two are not, and one is for analysis rather than for running an application.

Choosing the product: three questions

Analysis or application, relational or not, then scale and shape

Figure: A decision tree. If the job is analysing large datasets rather than serving an application, choose BigQuery. Otherwise, for relational data choose Spanner when it must be consistent across regions at global scale and Cloud SQL when it need not. For non-relational data choose Firestore for documents behind a web or mobile app, and Bigtable for single-keyed, high-throughput data.

Worked example (synthetic). A retailer's PostgreSQL inventory app serves one country and must keep its standby ready if a zone fails. Relational, not global: Cloud SQL, with its high availability option.

Here is the choice as three questions, asked in this order because each one removes products. First: is the job analysing large datasets rather than serving an application? If so, the answer is BigQuery, whose columnar storage is optimized for analytical queries. Second: is the data relational? If it is, the third question splits the two relational products — must it stay consistent across regions at global scale? Spanner offers exactly that; Cloud SQL is the managed relational database for when it need not, and its high availability option provides a standby virtual machine in another zone. If the data is not relational, ask what shape it has. Documents behind a web or mobile app point to Firestore, which also supports, but does not require, the Firebase platform. Large amounts of single-keyed data needing high throughput and low latency — time series, meter readings — point to Bigtable.

The three pairs the exam swaps

Similar names or similar data — different products

Figure. Three cards contrasting the product pairs candidates confuse: Bigtable against BigQuery, Cloud SQL against Spanner, and Bigtable against Firestore, each with the deciding difference.

Worked example (synthetic). A team proposes Bigtable for an app's user profiles because they are "big". The profiles are structured documents needing transactions — Google's Bigtable page itself points to Firestore.

Three pairs account for most wrong answers in this objective. Bigtable and BigQuery share half a name and nothing else: Bigtable serves large amounts of single-keyed data at low latency to an application, while BigQuery combines a data warehouse with analytic tools. Cloud SQL and Spanner are both relational; the difference is scale and consistency, because Spanner offers transactional consistency at global scale with automatic, synchronous replication, while Cloud SQL is the fully managed service for MySQL, PostgreSQL and SQL Server. And Bigtable against Firestore is settled by Google on the Bigtable page itself: if you must store highly structured objects in a document database, with support for ACID transactions and SQL-like queries, consider Firestore.

Choosing a storage option

Block storage under a VM, or object storage in a bucket

• Durable block storage survives the VM stopping, being deleted or failing
• Zonal disk: one zone. Regional disk: replicated across two zones
• Google now recommends Hyperdisk where the machine series supports it
• Cloud Storage class: chosen by how often the data will be read

Worked example (synthetic). A database VM's data disk must survive a zone outage, and nightly exports must be kept for audits nobody expects to run. A regional Persistent Disk for the first; Archive storage for the second.

The second objective lists two families of storage, and the first decision is which family. Block storage sits under a virtual machine, or VM. Google distinguishes durable block storage — for data you want to preserve after you stop, suspend or delete the VM, or even if it crashes — from temporary storage that is lost when the VM stops. Within durable storage, the guide names zonal and regional Persistent Disk: a disk created in only one zone is a zonal disk, and a disk replicated across two zones in the same region is a regional Persistent Disk. One thing has moved since the guide was written. Google now says Hyperdisk and Persistent Disk are the durable block storage offerings, that Persistent Disk is not available with the latest machine series, and that you should use Hyperdisk where your machine series supports it. The exam asks about Persistent Disk, so this deck teaches it — but in a real project, check the machine series first. The other family is Cloud Storage, where every object carries a storage class, a piece of metadata chosen by how often the data will be read.

Zonal against regional Persistent Disk

A regional disk writes to two zones, so a zone outage is survivable

Figure: One region containing two zones. In zone a, a virtual machine writes to a disk replica, which is synchronously replicated to a second replica in zone b. A zonal disk would have only the zone a replica.

Worked example (synthetic). Zone a has an outage. With a zonal disk the data waits for the zone; with a regional disk the replica in zone b can be accessed immediately.

This is the whole difference between the two disks the guide names. A zonal disk lives in one zone. A regional Persistent Disk keeps a replica in each of two zones in the same region, and Google says that if a primary zone becomes unavailable during an outage, the replica in the second zone can be accessed immediately. That is why regional disks are designed for workloads that need a lower recovery point objective and recovery time objective than restoring from snapshots. It has a cost, and Google names it: regional Persistent Disk is an option when write performance is less critical than data redundancy across multiple zones. And it has a limit worth remembering — disk replication only provides high availability of the disks. The virtual machine in the failed zone still has to come back somewhere else.

Creating the disk: types, defaults and flags

The console and gcloud do not default to the same disk type

Worked example (synthetic). An engineer scripts a disk with gcloud and omits the type flag, expecting what the console showed yesterday. The script creates a standard disk, not a balanced one.

Now the hands-on part. When you create a Persistent Disk you choose a type, and the names are what you type. Balanced Persistent Disk, written pd-balanced, offers performance suitable for most general-purpose applications. Solid-state drive Persistent Disk, pd-ssd, suits enterprise applications and high-performance databases needing lower latency. Standard Persistent Disk, pd-standard, suits large data processing workloads that primarily use sequential reads and writes. You create a zonal disk with the gcloud compute disks create command and a zone. For a regional disk the same command takes a region and a replica-zones flag naming the two zones where the replicas are located. The detail the exam likes is the default: if you create a disk in the Google Cloud console the default type is pd-balanced, but if you create it using the gcloud command-line tool or the Compute Engine interface for programs, the default is pd-standard. Leave the flag off in a script and you get the slower disk.

The four Cloud Storage classes

Colder class: cheaper to keep, dearer to read, longer to commit

Worked example (synthetic). Monthly analytics exports go to Nearline; quarterly compliance snapshots to Coldline; seven-year legal holds to Archive. Picking Archive for the monthly exports saves on storage and loses on every read.

The four classes are one ladder, and each step down trades the same three things. Standard storage is best for frequently accessed, hot data and for data stored only briefly, and it is what a bucket gets if you do not specify a default class. Nearline is ideal for data you read or modify on average once a month or less; Google calls it the better choice when slightly lower availability, a thirty-day minimum storage duration and costs for data access are acceptable trade-offs for lower at-rest storage costs. Coldline is for data read at most once a quarter, with a ninety-day minimum. Archive is for data you plan to access less than once a year, with higher costs for access and operations and a three-hundred-and-sixty-five-day minimum. One misconception to drop: colder does not mean slower to reach. Cloud Storage provides low latency access to data stored as Archive storage — there is no offline retrieval step.

Configuring the bucket: location, default class, and changing it

A new default class does not touch the objects already stored

Figure. Four cards: bucket location types (region, dual-region, multi-region) and that regions or dual-regions survive a zone loss; placing data near the VMs that use it; changing the default storage class with gcloud storage buckets update and the default-storage-class flag, which leaves existing objects unchanged; and changing existing objects by rewriting them or with a lifecycle SetStorageClass rule.

Worked example (synthetic). A team switches a bucket's default class to Coldline and expects last year's files to move. They do not: only new objects get Coldline until a lifecycle rule or a rewrite changes the old ones.

Two settings on the bucket itself finish the objective. The first is location. Cloud Storage offers a region, which is a specific geographic place such as Iowa; a dual-region, which is a specific pair of regions; and a multi-region, a large area containing two or more regions such as the United States. To withstand the risk of losing data within an individual zone, Google says to use regions or dual-regions — data stored in a region is redundant across multiple zones within it — and storing data in the same zone or region as the Compute Engine virtual machines that use it can provide better performance. The second is the default class, and it holds the trap this exam likes. You change it with the gcloud storage buckets update command and the default-storage-class flag. But changing the default storage class of a bucket does not affect any of the objects that already exist in the bucket. To move existing objects you either rewrite them or use Object Lifecycle Management, whose SetStorageClass action switches objects to another class — and when rules collide, the action that moves an object to the class with the lowest at-rest pricing takes precedence.

What this topic actually tests

Two choices, four discriminations

Analysis or application? BigQuery, or one of the four others. Relational at what scale? Spanner globally, Cloud SQL otherwise. Survive a zone? a regional disk replicates to two zones. How often is it read? that picks the class — and a new default class leaves existing objects where they are.

Close the topic on the four discriminations it tests. First, analysis or application: BigQuery is the warehouse for analytical queries, and the other four serve applications. Second, relational at what scale: Spanner offers transactional consistency at global scale, and Cloud SQL is the managed MySQL, PostgreSQL and SQL Server service when that is not needed; non-relational data goes to Firestore as documents or to Bigtable as single-keyed, high-throughput data. Third, must the disk survive a zone: a regional Persistent Disk replicates across two zones, a zonal disk does not — and remember the gcloud default disk type is pd-standard. Fourth, how often will the data be read: that picks Standard, Nearline, Coldline or Archive, and changing a bucket's default class does not move a single existing object.

Official sources for this topic

• Associate Cloud Engineer exam guide — Section 2
• Cloud SQL overview
• Spanner documentation
• Firestore overview
• Bigtable overview
• BigQuery overview
• Choose a disk type — Compute Engine
• About Persistent Disk — Compute Engine
• Storage classes — Cloud Storage
• Add a Persistent Disk to your VM — Compute Engine
• Create regional disks — Compute Engine
• Bucket locations — Cloud Storage
• Change the default storage class of a bucket — Cloud Storage
• Object Lifecycle Management — Cloud Storage