Managing Compute Engine resources

Study guide for Associate Cloud Engineer, Unit 4 · Topic 1. This is the topic's lecture in reading form — every slide's teaching, figures and worked examples, in order — followed by the official Google Cloud pages its claims rest on.

What the exam guide asks, quoted. Managing Compute Engine resources. Considerations include:

1. Remotely connecting to the instance
2. Viewing current running VM inventory (e.g., instance IDs, details)
3. Working with snapshots (e.g., create a snapshot from a VM, view snapshots, delete a snapshot, schedule a snapshot)
4. Working with images (e.g., create an image from a VM or a snapshot, view images, delete an image)

Managing Compute Engine resources

Get in, see what is there, keep a copy, make more of it

Get in — connect to an instance, even one with no external IP address. See what is there — the running inventory, and the ID that never changes. Keep a copy — snapshots, one-off and scheduled. Make more of it — images, image families, and what deprecating one does.

Unit 3 built the instances. This topic is about living with them, and its four objectives are the four things an engineer does to a fleet that already exists. First, getting in: remotely connecting to an instance, including one that has no external internet protocol (IP) address at all. Second, seeing what is there: the running virtual machine (VM) inventory, and which value identifies a VM for good. Third, keeping a copy: snapshots — creating one from a VM's disk, viewing and deleting them, and putting them on a schedule. Fourth, making more of it: images — creating one from a VM or from a snapshot, viewing and deleting them. Hold one distinction across the last two objectives, because Google states it outright and the exam leans on it: a snapshot is how you back up a disk, and an image is how you stamp out new VMs.

Remotely connecting to the instance

Which tool you use decides where the key lives

• Console SSH button: an ephemeral key, kept in the browser, gone in three minutes
• gcloud compute ssh: a persistent key, public half in project metadata
• Third-party clients (OpenSSH, PuTTY): you create and upload the key yourself
• Windows: RDP on TCP port 3389, with a Windows account and password

Worked example (synthetic). An engineer connects with the SSH button in the console, copies nothing, and the next morning tries to reach the same VM from a laptop terminal with plain ssh. There is no key to use: the console's key lived in the browser and expired in minutes.

Google's model for connecting to a Linux virtual machine (VM) is key-based secure shell (SSH): Compute Engine uses key-based SSH authentication to establish connections, and also supports certificate-based authentication for VMs that use OS Login, Google's operating system login service. What changes between tools is who makes the key and where it lives. If you use the Google Cloud console or the gcloud tool, Compute Engine performs the setup on your behalf; if you connect any other way, you must perform some configurations yourself. The console's SSH button creates an ephemeral key pair: your public and private SSH keys are stored in your browser session, and your SSH key has an expiry of three minutes. The gcloud compute ssh command instead creates a persistent SSH key pair — the public key is stored in project metadata, and the key doesn't have an expiry. Google describes gcloud compute ssh as a thin wrapper around ssh that takes care of authentication and of translating the instance name into an internet protocol (IP) address. Windows is different again: Compute Engine creates firewall rules that allow remote desktop protocol (RDP) access on transmission control protocol (TCP) port 3389, and you sign in with a Windows account and password, which you create if you don't have one yet.

Four ways in, and who manages the key

The same VM, reached four ways

Worked example (synthetic). A team scripts nightly maintenance over SSH. The console's key cannot serve it — it expires in minutes — so the script uses gcloud compute ssh, whose key persists.

Laid side by side, the four routes to a virtual machine (VM) over secure shell (SSH) or remote desktop differ in exactly the way the exam probes: who creates the key, where it is kept, and how long it lasts. The console's SSH button has Compute Engine create a key that lives in your browser session and expires after three minutes — ideal for a quick look, useless for anything repeated. gcloud compute ssh has Compute Engine create a persistent key, storing the public half in project metadata; if project metadata keys are blocked, it stores it in instance metadata instead. Its key doesn't have an expiry, and in Cloud Shell — a shell environment with the Google Cloud command-line tool already installed and your current project already set — you can often omit the project and zone flags entirely. Third-party clients such as OpenSSH or PuTTY need you to create the key and add it to the VM yourself, and they connect to the VM's external internet protocol (IP) address. And Windows VMs are reached over remote desktop protocol (RDP) with a Windows account and password. One more rule sits over the Linux routes: if you use OS Login — the operating system login service — metadata SSH keys are disabled — that was Unit 3's topic, and it changes where every one of these keys is honored.

Connecting to a VM with no external IP address

IAP opens the tunnel — after IAM says yes

Figure. An engineer runs gcloud compute ssh with the tunnel-through-iap flag. Identity and Access Management checks whether the engineer holds the IAP-secured Tunnel User role. Identity-Aware Proxy then forwards the SSH connection over an encrypted tunnel into the VPC network, whose firewall must admit IAP's address range 35.235.240.0/20, to a VM that has only an internal IP address, on port 22.

Worked example (synthetic). A bank's VMs have no external IP addresses by policy. Engineers still reach them with one flag, because IAP forwards the SSH connection to each VM's internal address.

A virtual machine (VM) with no external internet protocol (IP) address is not unreachable — it is reachable through Identity-Aware Proxy (IAP). Google describes this as connecting to a VM through its internal IP address using IAP transmission control protocol (TCP) forwarding, which lets you establish an encrypted tunnel over which you can forward secure shell (SSH), remote desktop protocol (RDP) and other traffic. The command is the one you already know with one flag added: gcloud compute ssh with --tunnel-through-iap. Read the picture left to right. The engineer's request reaches identity and access management (IAM) first, because IAP only tunnels for users granted the right role on the project — the IAP-secured Tunnel User. Then IAP forwards the connection into the network. Its traffic arrives from a fixed range, 35.235.240.0/20, which contains all IP addresses IAP uses for TCP forwarding — so a firewall rule must allow ingress from that range on port 22 for SSH, or 3389 for RDP. The VM itself needs no external IP address. Note also that the console's SSH-in-Browser only uses IAP if the VM doesn't have an external IP address.

Making IAP the only way in

Adding a door does not close the old ones

Figure. Three cards: admit IAP's address range on the SSH and RDP ports; grant the IAP-secured Tunnel User role only to those who need it; and disable or delete the default-allow-ssh and default-allow-rdp rules, which admit all IP addresses.

Worked example (synthetic). A security team sets up IAP for a project on the default network and declares SSH locked down. A penetration test still connects directly: the default network's allow-SSH rule was never removed.

Setting up IAP is two steps; making it the only route in is three, and the third is the one teams forget. Step one: a firewall rule admitting IAP's range. Step two: the IAP-secured Tunnel User role for the people who should have it. Step three: close the doors that were already open. Google's own reference for gcloud compute ssh says the default network comes preconfigured to allow ssh access to all VMs, and the IAP guide is explicit about why that matters — the default-allow-ssh and default-allow-rdp rules allow secure shell (SSH) and remote desktop protocol (RDP) connections from all internet protocol (IP) addresses, not only from IAP. Google's advice is to consider disabling or deleting those rules if you want to prevent direct access. For Windows specifically, Google says that when connecting from anywhere over the public internet, it's best to enable Identity-Aware Proxy (IAP) transmission control protocol (TCP) forwarding for your project.

Viewing the running VM inventory

List to find them; describe to know one; the ID never changes

• Console: the VM instances page, with columns you can change
• gcloud compute instances list: every instance in the project
• gcloud compute instances describe: every setting of one instance
• The VM's ID is the immutable way to reference it

Worked example (synthetic). An engineer inherits a project and is asked what is running. One list command answers it; one describe command answers what a particular VM has attached.

The second objective is seeing what is running, and it has a list half and a detail half. The list half: gcloud compute instances list displays all Compute Engine instances in a project. It narrows in the usual ways — with --zones, only resources from the given zones are queried, and --filter applies a Boolean filter expression to each resource item to be listed. In the console, the VM instances page shows a default set of columns, and you can change which columns appear. The detail half: to view the details of a virtual machine (VM), use gcloud compute instances describe, which displays all data associated with the instance. Google spells out why details matter — they show configuration and status, such as attached disks, creation timestamp, machine type, and the ID of the VM, which you can use to reference the VM using an immutable value. That last phrase is the exam's point: a name identifies a VM today, but the ID is the value that never changes.

Which command answers which inventory question

Five questions, five commands

Worked example (synthetic). An auditor asks for the machine type and attached disks of one VM, then separately for the packages installed on it. The first is describe; the second is VM Manager's inventory — describe cannot see inside the guest.

Here is the objective as the exam asks it: a question, and the command that answers it. Every VM in the project is gcloud compute instances list. The same list restricted to some zones adds --zones, and the same list restricted by a condition adds --filter; Google's own reference also shows --uri, to list the uniform resource identifiers (URIs) of all instances instead. Everything about one VM is gcloud compute instances describe, with its zone. The last row is the one that catches people. None of these commands can tell you what software is installed inside the guest operating system (OS). That is a different inventory — after setting up VM Manager on your virtual machines (VMs), you can view OS inventory data and vulnerability reports. The instance inventory is Compute Engine's view of the machine; the OS inventory is the machine's view of itself.

Working with snapshots

Incremental disk backups — even from a running VM

• Create: gcloud compute snapshots create, from a disk — attached and running is fine
• Standard snapshots are incremental and geo-redundant
• View: snapshots list (filter by source disk), snapshots describe
• Delete: snapshots delete — which may not free all of its data

Worked example (synthetic). A database VM cannot be stopped for backups. Its data disk is still snapshotted every night, because a snapshot can be taken while the disk is attached to a running VM.

The third objective is the disk backup. Standard snapshots incrementally back up data on a disk, and they provide geo-redundant backups of a single disk regardless of whether the disk is attached to a running virtual machine (VM) — Google's create page says directly that you can create snapshots from disks even while they are attached to running instances. If you do pause the application for consistency, resume it only after the snapshot resource reaches the UPLOADING status. The command is gcloud compute snapshots create, and Google recommends it over the older gcloud compute disks snapshot because it supports more features. Viewing is gcloud compute snapshots list for the project, the same command with --filter for one disk's snapshots, and gcloud compute snapshots describe for one snapshot. Deleting is gcloud compute snapshots delete — with a subtlety the exam likes. Snapshots are incremental, so each contains only what changed since the previous one; because later snapshots may depend on data stored in an earlier one, deleting a snapshot does not necessarily delete all the data on the snapshot. And when a whole VM with several disks must be backed up at once, the answer is a machine image, not a pile of snapshots.

Three kinds of snapshot

Only one of them dies with its disk

Worked example (synthetic). A team snapshots a disk with instant snapshots, deletes the disk during a cleanup, and finds its backups gone with it. Standard or archive snapshots would have survived.

All three types capture a disk at a point in time; what separates them is what happens later. Standard snapshots are the default backup. Archive snapshots suit retention for many months or years when you rarely need to access them — and deleting an archive snapshot is irreversible. Instant snapshots are the trap: an instant snapshot of a disk only exists until the source disk is deleted. Standard and archive snapshots aren't deleted with the source disk, so if you want to keep a backup after deleting the disk itself, it must be one of those two. The exam's version of this is a cleanup that deletes a disk and the question of which backups are still there.

Scheduling snapshots

A schedule is a resource policy you attach to a disk

Figure: A flow: create the snapshot schedule as a resource policy in the same region as the disk; attach it to the disk with add-resource-policies; snapshots are then taken hourly, daily or weekly and kept according to the retention policy. To delete the schedule, first detach it from the disk; once deleted, the snapshots it made are kept permanently.

Worked example (synthetic). An engineer deletes an obsolete snapshot schedule expecting its snapshots to go with it. They remain, now kept permanently, until someone deletes them by hand.

A snapshot schedule regularly and automatically backs up disks, and Google calls using them a best practice. Mechanically, creating one creates a resource policy that you can apply to one or more disks — so it is two steps. First, gcloud compute resource-policies create snapshot-schedule, and the schedule must be in the same region where the disk resides; you choose hourly, daily or weekly frequency, and with the gcloud tool or the console you must always set a retention policy. Second, attach it with gcloud compute disks add-resource-policies. Changing a schedule is forward-only: only snapshots created after the update use the new values. Removing one has two rules worth memorizing. If the schedule is attached to a disk, detach it first, then delete it. And deleting a schedule does not delete its work — all auto-generated snapshots associated with it are kept permanently. The same instinct applies when the disk goes away: by default, keep-auto-snapshots keeps every snapshot the schedule created, regardless of the retention window.

Working with images

An image is a saved state to build new VMs from

• Sources: a disk, a snapshot, another image, or a file in Cloud Storage
• gcloud compute images create — from a running VM's disk only with --force
• Image families: point automation at the family, not a version
• View with images list; delete with images delete

Worked example (synthetic). A team hardens one VM's boot disk, captures it as an image in the family web-base, and every new web VM is created from the family — so next month's rebuilt image is picked up without a script change.

Images answer a different question from snapshots. Custom images are ideal when you have brought a boot disk or image to a certain state and need to save that state for creating virtual machines (VMs) — and Google's own page adds, if you want incremental backups of disk data, use snapshots instead. You can create custom images from source disks, from images, from snapshots, or from images stored in Cloud Storage, with gcloud compute images create. The trap is the running VM. By default, you cannot create an image from a disk attached to a running instance; the --force flag lets you, but Google's preferred preparation is to stop the VM so it shuts down and stops writing to the disk. If you don't choose a storage location, Compute Engine stores the image in the multi-region closest to the source. Viewing is gcloud compute images list, which displays all images in a project, and deleting is gcloud compute images delete. Then there are image families. They let you group related images so that automation references the latest version while keeping the ability to roll back — you reference the family name instead of updating an image name at intervals.

Where an image can come from

One command, four sources

Worked example (synthetic). An engineer's image create from a production VM's boot disk fails. Rather than add --force, the team images last night's snapshot of the same disk and never touches the running VM.

The objective's two named cases — an image from a virtual machine (VM), and an image from a snapshot — are two flags on one command. From a VM, you name its disk with --source-disk and its zone. Here the reference is blunt: by default, image creation fails when it is created from a disk attached to a running instance; add --force to override, or stop the VM first. From a snapshot, you name it with --source-snapshot, which sidesteps the running VM entirely. From another image, --source-image. And --family places the new image in an image family: when an instance or disk is created from a family, the latest non-deprecated image in the family is used.

Image families and deprecation

Deprecate the newest image and the family steps back

Figure: Image deprecation states in order: ACTIVE, where the family points if it is the newest image; DEPRECATED, which the family skips but which can still create VMs with a warning; OBSOLETE, no longer available for use; and DELETED, where use returns an error. A dashed arrow notes a deprecated image cannot return to ACTIVE.

Worked example (synthetic). A new web-base image ships with a bug. The team deprecates it; new VMs from the web-base family immediately get last week's image again — the family's roll-back, with no script edited.

Image families and deprecation work together, and this is where the roll-back promise is kept. Image families point to the most recent and active image in a family. Mark an image DEPRECATED and it can still be used to create a virtual machine, but image families no longer point to it — even if it is the most recent image in the family. So deprecating a bad new release makes the family fall back to the previous good one, and every automation that references the family name recovers without an edit. Further down, OBSOLETE means the image is no longer available for use, and a DELETED image returns an error. Two operational notes. Deprecation is one-way: once you deprecate an image, you can't set it back to ACTIVE. And gcloud compute images list hides deprecated images unless you add --show-deprecated.

What this topic actually tests

Four discriminations, four objectives

Which key, and how long? console keys expire; gcloud keys persist; no external IP means IAP. Name or ID? the ID is immutable. Which snapshot survives? standard and archive outlive the disk; instant does not; a deleted schedule's snapshots stay. Snapshot or image? snapshots back up; images build VMs, and a deprecated image drops out of its family.

Close on the four discriminations this topic rewards. Connecting: the console's key lives in the browser for three minutes, gcloud compute ssh's key persists in project metadata, and a VM with no external internet protocol (IP) address is reached through Identity-Aware Proxy (IAP) with a firewall rule for 35.235.240.0/20 and the IAP-secured Tunnel User role. Inventory: list to find virtual machines (VMs), describe to know one, and the ID is the immutable reference. Snapshots: they can be taken from running VMs, standard and archive snapshots outlive their disk while instant snapshots do not, schedules are resource policies you attach and must detach before deleting, and a deleted schedule's snapshots are kept. Images: they save a state for building VMs, creation from a running VM's disk fails without --force, and deprecating an image makes its family step back to the previous one. The next topic takes the same day-2 view to Google Kubernetes Engine.

Official sources for this topic

• Associate Cloud Engineer exam guide — Section 4
• Create custom images — Compute Engine
• About SSH connections — Compute Engine
• gcloud compute ssh — Google Cloud SDK reference
• Connect to Windows VMs using RDP — Compute Engine
• gcloud compute instances list — Google Cloud SDK reference
• View VM details — Compute Engine
• gcloud compute instances describe — Google Cloud SDK reference
• About disk snapshots — Compute Engine
• Create standard snapshots — Compute Engine
• Manage disk snapshots — Compute Engine
• gcloud compute images list — Google Cloud SDK reference
• gcloud compute images delete — Google Cloud SDK reference
• Image families best practices — Compute Engine
• Connect to Linux VMs — Compute Engine
• Connect to Linux VMs using Identity-Aware Proxy — Compute Engine
• Using IAP for TCP forwarding — Identity-Aware Proxy
• gcloud compute start-iap-tunnel — Google Cloud SDK reference
• View operating system details — Compute Engine
• Create snapshot schedules — Compute Engine
• Manage snapshot schedules — Compute Engine
• gcloud compute images create — Google Cloud SDK reference
• Deprecate a custom image — Compute Engine