Managing Google Kubernetes Engine resources

Study guide for Associate Cloud Engineer, Unit 4 · Topic 2. This is the topic's lecture in reading form — every slide's teaching, figures and worked examples, in order — followed by the official Google Cloud pages its claims rest on.

What the exam guide asks, quoted. Managing Google Kubernetes Engine resources. Considerations include:

1. Viewing current running cluster inventory (e.g., nodes, Pods, Services)
2. Configuring Google Kubernetes Engine to access Artifact Registry
3. Working with node pools (e.g., add, edit, or remove a node pool)
4. Working with Kubernetes resources (e.g., Pods, Services, Statefulsets)
5. Managing Horizontal and Vertical autoscaling configurations

Managing Google Kubernetes Engine resources

Day two: the cluster exists, and now it has to be run

See what is running — clusters, node pools, Pods, Services. Let nodes pull images from Artifact Registry. Change the nodes — add, edit, resize or remove a node pool. Manage the workloads — Pods, Services, StatefulSets. Let it scale itself — horizontally, vertically, and at the node level.

Unit 3 built a Google Kubernetes Engine cluster and deployed a first application to it. This topic is what happens on every day after that, and the exam guide lists five things. Seeing what is running: the clusters, the node pools, the Pods and the Services. Letting the nodes pull container images from Artifact Registry. Working with node pools — adding one, editing one, resizing one, removing one. Working with the Kubernetes resources themselves — Pods, Services and StatefulSets. And managing autoscaling, horizontally and vertically. Almost every item on the exam here is a command, or a consequence of running one, so this deck is built around the commands Google documents and the one or two surprises each of them carries.

Viewing the running cluster inventory

gcloud answers questions about clusters; kubectl answers questions inside one

• Clusters in a project: gcloud container clusters list, and describe for one
• Node pools in a cluster: gcloud container node-pools list --cluster
• Pods and Services: kubectl get pods, kubectl get service
• Which node a Pod is on: kubectl get pods --output=wide

Worked example (synthetic). An on-call engineer is paged about a slow application. gcloud container clusters list finds the cluster, kubectl get pods --output=wide shows three Pods crowded onto one node, and kubectl get service confirms the external address users are hitting.

The first objective is inventory, and the organizing rule is simple: the Google Cloud command-line interface, gcloud, answers questions about clusters, and kubectl answers questions inside one. At the project level, Google's page is direct — to list all clusters in a project in any region or zone, run gcloud container clusters list, and to view a specific cluster, run gcloud container clusters describe with the cluster's name. One level down, the node pools: to check the status of all node pools, including those not yet fully provisioned, run gcloud container node-pools list with the cluster flag. Inside the cluster, kubectl takes over. Google's quickstart inspects the running Pods with kubectl get pods and the Service with kubectl get service. To see every application, run kubectl get with the controller type — deployments, statefulsets, or another controller object type. And when the question is which node a Pod landed on, Google's workload separation page lists Pods with the output flag set to wide to find the names of the nodes. The console has an equivalent for the workload side, the Workloads page, where Google also shows autoscaling events and what caused them.

The question, and the command that answers it

Pick the tool by the level of the question

Worked example (synthetic). A reviewer asks for every node pool across the organization's clusters. The engineer runs clusters list first — node-pools list needs a --cluster and answers for one cluster at a time.

Here is the inventory as a lookup table, and the column worth reading is the last one. Project-level questions — which clusters exist — are gcloud container clusters list, which Google says lists all clusters in any region or zone. Cluster-level questions are still gcloud: describe for one cluster's configuration, and node-pools list for its node pools, which takes the cluster flag because it answers for one cluster at a time. Everything inside the cluster is kubectl: kubectl get with deployments or statefulsets for the applications, kubectl get pods with wide output for where each Pod runs, and kubectl get service for the address a Service exposes. When an exam option offers kubectl for a project-level question, or gcloud for which node a Pod is on, the level is wrong and so is the option.

Letting GKE pull from Artifact Registry

The node's service account is the identity that pulls the image

• Nodes run as the Compute Engine default service account unless you choose otherwise
• Same project: grant that account roles/artifactregistry.reader
• Different project: grant Artifact Registry permissions to the node service account
• Access scopes are fixed on existing nodes; public repositories need no authentication

Worked example (synthetic). A new cluster's Pods sit in an image-pull error. The organization disabled the automatic Editor grant, so the default node service account has no Artifact Registry role — granting it the Reader role in the repository's project fixes every node at once.

The second objective is configuring Google Kubernetes Engine to access Artifact Registry, and the whole objective turns on one fact: the identity that pulls an image is the node's service account. Google states the default directly — the Compute Engine default service account is the identity for nodes. Google Kubernetes Engine can pull images directly from Docker repositories, and with the defaults plus the Artifact Registry Reader role on that account, it can pull from Artifact Registry repositories in the same Google Cloud project. Two conditions change the answer. If the organization has disabled the automatic grant of the basic Editor role, Google says to grant the Compute Engine default service account the Artifact Registry Reader role, roles slash artifactregistry dot reader. And if the cluster is in a different project from the repository, you grant Artifact Registry permissions to the service account that the nodes use. Two traps close the objective. Google says you cannot change access scopes on existing nodes, so a scope problem is fixed on new nodes, not edited on old ones. And if the default account already holds Editor, Google recommends replacing it with less permissive roles rather than relying on it. Public Artifact Registry repositories, finally, don't require authentication at all.

Why an image pull fails, and the fix for each case

Follow the identity: which account, which project, which role

Figure: A decision flow for a Pod that cannot pull its image. A public repository needs no authentication. A private repository in the same project needs the node service account granted the Artifact Registry Reader role; one in a different project needs Artifact Registry permissions granted to the node service account there. A Kubernetes workload outside Google Cloud uses Workload Identity Federation with Kubernetes.

Worked example (synthetic). A platform team keeps shared base images in a central project. Each application cluster's node service account is granted Artifact Registry permissions in that central project — nothing is copied into the application projects.

Drawn as a decision, the objective has three questions, and each one is about the identity doing the pull. First, is the repository public? Google says public Artifact Registry repositories don't require authentication, so nothing needs granting. If it is private, is it in the same project as the cluster? Then the fix is the Reader role on the node service account, which Google names for the case where the automatic Editor grant is disabled. If it is in another project, Google's instruction is to grant Artifact Registry permissions to the service account the nodes use. The third branch is for Kubernetes workloads that are not on Google Cloud at all: Google points them to Workload Identity Federation with Kubernetes. What the figure leaves out is as important as what it shows — there is no branch that edits access scopes on running nodes, because Google says you cannot.

Working with node pools

Add, edit, resize, remove — and resize is not a node-pools command

Worked example (synthetic). An engineer types gcloud container node-pools resize from memory and gets an error. Google's page resizes a pool with gcloud container clusters resize and the --node-pool flag.

The third objective is node pools, and Google defines the unit first: a node pool is a group of nodes within a cluster that all have the same configuration. Everything else follows from that. To add capacity with a different configuration, you create a pool — Google's command is gcloud container node-pools create, with the cluster named. To change the machine type, disk type or disk size of an existing pool, Google says you can modify the node pool's configured machine type, disk type, and disk size, with gcloud container node-pools update. To change how many nodes a pool has, the command is the one people get wrong: Google resizes a cluster's node pools with gcloud container clusters resize, naming the pool with the node-pool flag. It is a clusters command, not a node-pools one. And to remove a pool, gcloud container node-pools delete — with Google's warning that deleting a node pool deletes the nodes and all running workloads. One scope note: Google says everything on its node pool page except upgrades applies specifically to Standard node pools. Autopilot manages its nodes for you.

Node pools inside one cluster

Each pool is one configuration; a new configuration is a new pool

Figure. A Google Kubernetes Engine Standard cluster containing two node pools. The default pool holds general-purpose nodes; a second pool holds Spot VM nodes. Within each pool the nodes are alike. Requests from kubectl and gcloud reach the cluster through its control plane endpoint. The callout says a different machine type is a different node pool, not an edited node.

Worked example (synthetic). A cluster needs cheap capacity for batch jobs. Rather than changing the existing nodes, the team adds a batch-pool of Spot VMs beside the default-pool and schedules the batch jobs onto it.

This is the containment the objective rests on. The outer box is one Standard cluster, reached through its control plane endpoint. Inside it are node pools, and inside each pool the nodes are alike, because Google defines a node pool as nodes that all have the same configuration. The first pool is the default one — Google says it consists of three nodes in each of the cluster's compute zones, with the default node image and a general-purpose machine type. The second pool exists because some workloads need something different, and Google's examples are exactly that: a node pool with local solid-state drives, a minimum CPU platform, Spot VMs, a different node image or different machine types. So when a scenario says some workloads need a different kind of node, the answer is a new pool beside the old one, not a change to individual nodes.

What resizing and deleting actually do

Both drain nodes; neither should fight the autoscaler

Figure. Five cards. Growing a pool creates nodes with the same configuration. Shrinking drains nodes gracefully, and a particular node can be drained first. Deleting a pool drains every node and reschedules Pods only where another pool has room. A pool with cluster autoscaler should not also be resized by hand. Autopilot clusters resize themselves.

Worked example (synthetic). Before a sale, an engineer manually resizes a pool that already has cluster autoscaler enabled. Google warns against exactly this: the manual and automatic sizes can fight, leaving the pool unstable.

The commands are short; their consequences are what the exam tests. When you grow a pool, Google says new node instances are created using the same configuration as the existing instances. When you shrink it, Google Kubernetes Engine gracefully terminates the removed nodes by draining them, and if a particular node must go, Google's advice is to drain that node first and then resize. Deleting a pool is the larger version: Google says Google Kubernetes Engine drains all the nodes in the pool, deleting and rescheduling all Pods — which only works if some other pool has room for them. Two warnings complete the picture. Do not use both cluster autoscaler and manual resize commands simultaneously on a node pool, because Google says the interaction can result in an unstable or incorrect node pool size. And none of this applies to Autopilot, whose clusters automatically resize based on the number of Pods.

Working with Pods, Services and StatefulSets

Controllers run Pods; Services give them a stable address

• A Deployment runs multiple replicas of Pods across the cluster's nodes
• kubectl scale --replicas changes the replica count instantly
• A Service groups Pod endpoints behind one stable IP address
• A StatefulSet gives each Pod a persistent identity and its own storage

Worked example (synthetic). A shop runs its web tier as a Deployment behind a Service and its database as a StatefulSet. Scaling the web tier is kubectl scale --replicas; the database keeps each replica's name and disk when its Pod moves.

The fourth objective is the Kubernetes resources themselves, and the organizing idea is that you rarely manage a bare Pod. Google describes a Deployment as a Kubernetes application programming interface object that lets you run multiple replicas of Pods distributed among the nodes in a cluster, and Google's own vertical autoscaling page notes that features like it work only with workloads managed by a controller — Deployments, StatefulSets and the like — not with standalone Pods. Scaling a controller is quick: Google says the kubectl scale command lets you instantaneously change the number of replicas, by setting the replicas flag. Services solve a different problem. The idea of a Service, in Google's words, is to group a set of Pod endpoints into a single resource, and with a Service you get a stable internet protocol address that lasts for the life of the Service, even as the IP addresses of the member Pods change. StatefulSets are for the workloads where Pods are not interchangeable: Google says StatefulSets represent a set of Pods with unique, persistent identities and stable hostnames that Google Kubernetes Engine maintains regardless of where they are scheduled, and each replica gets its own PersistentVolumeClaim.

The Service types, by who reaches them

ClusterIP is the default; each type after it extends the one before

Worked example (synthetic). An internal reporting API needs to be reachable only from other Pods in the cluster. The default ClusterIP is enough; a LoadBalancer would expose it for no reason.

Services come in types, and Google defines each one by who sends the request and where it goes. ClusterIP is the default: internal clients send requests to a stable internal internet protocol, or IP, address. NodePort: clients send requests to the IP address of a node on the nodePort values the Service specifies. LoadBalancer: clients send requests to the IP address of a network load balancer. Google notes that NodePort is an extension of ClusterIP, and LoadBalancer an extension of NodePort, so each type keeps what the one before it had. The fourth type matters for StatefulSets. A headless Service gives you a Pod grouping without a stable IP address, and Google says a StatefulSet is paired with a matching headless Service when it is initialized.

Deployment or StatefulSet

Interchangeable replicas, or replicas that keep who they are

Figure. Four cards comparing a Deployment, whose replicas are interchangeable and scaled with kubectl scale, with a StatefulSet, whose Pods keep a unique identifier and hostname and each get their own PersistentVolumeClaim. The last card says to remove an unhealthy Pod before scaling a StatefulSet.

Worked example (synthetic). A three-replica database StatefulSet has one Pod crash-looping. The engineer lists the Pods, deletes the unhealthy one, lets it return, and only then scales to five.

The last part of the objective is telling a Deployment from a StatefulSet, and Google's sentence for it is short: Pods in StatefulSets are not interchangeable — each Pod has a unique identifier that is maintained no matter where it is scheduled. A Deployment's replicas are the opposite; any one of them can serve a request. Storage follows from identity, since each StatefulSet replica gets its own PersistentVolumeClaim. To look at a StatefulSet, Google's command is kubectl get statefulset with the yaml output, which shows its live configuration. And StatefulSets carry one operational warning worth memorizing. If one appears unhealthy, Google says to get the list of Pods, see which are unhealthy, and remove the unhealthy Pod with kubectl delete pod — because attempting to scale a StatefulSet while it is unhealthy may cause it to become unavailable.

Horizontal and vertical Pod autoscaling

Horizontal changes how many Pods; vertical changes how big each one is

• Horizontal Pod autoscaler: more or fewer Pods, on CPU, memory or custom metrics
• kubectl autoscale creates one for average CPU; set min and max replicas
• Vertical Pod autoscaling: adjusts container requests and limits from usage
• Don't run both on CPU or memory — use multidimensional Pod autoscaling instead

Worked example (synthetic). A web tier's traffic doubles at lunch, so it gets a horizontal Pod autoscaler. A batch worker is always the same count but its memory requests were guessed, so it gets vertical Pod autoscaling instead.

The fifth objective is autoscaling, horizontally and vertically, and the two are answers to different questions. Google says the horizontal Pod autoscaler, or HPA, changes the shape of a workload by automatically increasing or decreasing the number of Pods, in response to CPU or memory consumption or to custom metrics. You can create one in the console, with kubectl apply, or, for average CPU only, with the kubectl autoscale command, and its manifest sets the minimum and maximum number of replicas the Deployment can scale between. To see the ones in a cluster, run kubectl get hpa. The vertical Pod autoscaler, or VPA, changes the size of each Pod rather than the count: Google says vertical Pod autoscaling automatically adjusts container resource requests and limits based on historical usage and real-time demand, and it is enabled by default in Autopilot clusters. The rule that joins them is a limitation Google states plainly: don't use the horizontal Pod autoscaler together with the vertical Pod autoscaler on CPU or memory. You can combine them on other metrics, and for the CPU and memory case Google points to multidimensional Pod autoscaling.

The vertical autoscaler's update modes

Recommend only, set once, or keep adjusting

Worked example (synthetic). A team wants to see what vertical Pod autoscaling would recommend before trusting it with production. It sets updateMode: "Off", reads the recommendations for a week, and only then switches modes.

The vertical autoscaler's behaviour is set by one field, updateMode, and the exam likes to ask which mode does what. Off: Google says vertical Pod autoscaling doesn't automatically apply any changes to a Pod, though you can still view its recommendations. Initial: it only assigns resource requests on Pod creation and never changes them later. Recreate: it evicts a Pod if it needs to change the Pod's resource requests. Auto is the default mode if the field is not specified, and Google says it behaves the same as Recreate — which is why a vertical Pod autoscaler added with no mode set will restart Pods. The newest mode, InPlaceOrRecreate, is in Preview, and aims to reduce disruption by updating Pod resources without re-creating the Pod when it can. One timing detail: in Auto or Recreate mode, Google says updates usually happen after a Pod is at least twenty-four hours old, to prevent frequent restarts.

Three autoscalers, three different things they change

Pods out, Pods up, and nodes to hold them

Figure: A flow chart. Changing load drives the horizontal Pod autoscaler, which changes the number of Pods. Usage history drives vertical Pod autoscaling, which changes each Pod's requests. Both feed the Pods' resource requests, which drive the cluster autoscaler to change the number of nodes between a minimum and maximum per node pool.

Worked example (synthetic). Pods sit Pending after a traffic spike although the nodes' actual CPU use is low. The cluster autoscaler still adds nodes, because it decides from the Pods' requests, not from utilization.

The objective says horizontal and vertical, and a third autoscaler sits underneath both, so it is worth drawing all three. The horizontal Pod autoscaler changes the number of Pods. Vertical Pod autoscaling changes each Pod's requests. Both of those change the same thing downstream — the resource requests the cluster has to find room for — and that is what the cluster autoscaler watches. Google says the cluster autoscaler in Google Kubernetes Engine automatically resizes the number of nodes in a node pool, based on the demands of your workloads, and that it makes its decisions based on the resource requests of the Pods rather than actual resource utilization. You configure it per node pool with a minimum and maximum size — the enable-autoscaling flag with min-nodes and max-nodes — and Google notes that min-nodes counts per zone. So a Pending Pod on a lightly used cluster is not a contradiction: the requests are what the cluster autoscaler reads.

What this topic actually tests

Five objectives, five discriminations

gcloud or kubectl? the level of the question decides. Which identity pulls the image? the node's service account. Resize or new pool? count is clusters resize --node-pool; a new configuration is a new pool. Deployment or StatefulSet? interchangeable or not. How many, how big, or how many nodes? HPA, VPA, cluster autoscaler — and never HPA with VPA on CPU or memory.

Close on the five discriminations the exam uses. First, gcloud or kubectl: gcloud for clusters and node pools, kubectl for everything inside a cluster. Second, image pulls: the identity is the node's service account, so the fix is a role on that account in the repository's project. Third, node pools: the node count changes with gcloud container clusters resize and the node-pool flag, and a different kind of node means a new pool, because every node in a pool shares one configuration. Fourth, workloads: a Deployment's replicas are interchangeable, a StatefulSet's are not, and a Service gives either a stable address. Fifth, autoscaling: the horizontal Pod autoscaler changes how many Pods, vertical Pod autoscaling changes how big each one is, and the cluster autoscaler changes how many nodes hold them — reading requests, not utilization. Google's one hard rule across them is not to run the horizontal and vertical autoscalers together on CPU or memory.

Official sources for this topic

• Associate Cloud Engineer exam guide — Section 4
• Managing clusters — GKE
• Add and manage node pools — GKE
• Deploy an app in a container image to a GKE cluster
• Scale an application — GKE
• Configure workload separation in GKE
• About horizontal Pod autoscaling — GKE
• Integrate with Google Kubernetes Engine — Artifact Registry
• About node pools — GKE
• Configure horizontal Pod autoscaling — GKE
• Configure vertical Pod autoscaling — GKE
• Understand Kubernetes Services — GKE
• About StatefulSets — GKE
• Deploy a stateful application — GKE
• About vertical Pod autoscaling — GKE
• Resize a Standard cluster — GKE
• About cluster autoscaler — GKE
• Autoscale a cluster — GKE