Topic 2 of 10
Operating GKE
The exam guide devotes half a dozen items to day-to-day GKE operation: viewing the cluster inventory, configuring access to Artifact Registry, working with node pools, with Kubernetes resources such as Pods, Services, and StatefulSets, managing horizontal and vertical Pod autoscaling, Autopilot resource requests, and traffic splitting.
Contents
Viewing the cluster inventory
Google separates two kinds of tools: Google Cloud tools (the console and gcloud container) to manage the cluster, and Kubernetes tools (kubectl) for what happens inside it. In the console, the GKE page shows clusters, nodes, and workloads.
gcloud container clusters list gcloud container clusters get-credentials CLUSTER --location=us-central1 kubectl get nodes kubectl get pods --all-namespaces kubectl get services kubectl describe pod POD_NAME
Each node carries the cloud.google.com/gke-nodepool label with its node pool's name, useful for filtering.
Access to Artifact Registry
GKE pulls images directly from Artifact Registry Docker repositories. The key is the nodes' identity:
- By default nodes use the Compute Engine default service account.
- If that account no longer has the Editor role (the
iam.automaticIamGrantsForDefaultServiceAccountsconstraint is enforced by default in organizations created since May 3, 2024), grant it Artifact Registry Reader (roles/artifactregistry.reader). - With that, GKE pulls images from repositories in the same project. For another project, pushing images, or a user-provided service account, configure access explicitly.
- Access scopes on existing nodes can't be changed.
Google recommends replacing the default account's Editor role with less permissive roles, using Policy Simulator to see the impact.
Node pools
A node pool is a group of nodes in a Standard cluster that share the same configuration. In Autopilot you don't configure them: GKE manages the nodes.
- The default pool has three nodes in each of the cluster's zones, with the
cos_containerdimage. - You add pools with other machines, Local SSD, Spot VMs, or another image for Pods with different needs, and steer Pods with
nodeSelectoror taints. - You can't configure a single node: any change affects every node in the pool.
- Each pool is created, updated, resized, and deleted independently; by default it uses the control plane's version.
gcloud container node-pools create highmem-pool --cluster=CLUSTER \ --location=us-central1 --machine-type=n2-highmem-4 --service-account=SA_EMAIL gcloud container clusters resize CLUSTER --node-pool=highmem-pool --num-nodes=2 --location=us-central1 gcloud container node-pools update highmem-pool --cluster=CLUSTER --location=us-central1 ... gcloud container node-pools delete highmem-pool --cluster=CLUSTER --location=us-central1
Cluster autoscaler
It automatically adjusts a pool's node count between a minimum and a maximum. It works from Pods' resource requests, not their actual usage:
- If there are Pods that don't fit on any node (unschedulable), it adds nodes.
- If nodes are underutilized and all Pods fit on fewer nodes, it removes nodes down to the minimum.
--min-nodes/--max-nodesare per zone;--total-min-nodes/--total-max-nodesare for the whole pool.- Removing nodes reschedules Pods: Google recommends controllers with multiple replicas, such as a Deployment.
- In Autopilot you don't need it: nodes are provisioned and scaled automatically.
gcloud container node-pools update default-pool --cluster=CLUSTER --location=us-central1 \ --enable-autoscaling --total-min-nodes=3 --total-max-nodes=12
Kubernetes resources
| Resource | What for |
|---|---|
| Pod | The smallest unit: one or more containers sharing network and storage |
| Deployment | Stateless applications with replicas, gradual updates, and rollback |
| StatefulSet | Pods with persistent identity, stable hostnames, and one persistent volume per replica. Paired with a headless Service |
| Service | A stable access point to a group of Pods |
In a StatefulSet, replica IPs can change if GKE reschedules a Pod: use Services instead of fixed IPs. For stateless applications, Google says to use Deployments.
Service types
| Type | How clients reach it |
|---|---|
| ClusterIP (default) | Internal clients, at a stable internal IP |
| NodePort | At a node's IP on a specific port. Extends ClusterIP |
| LoadBalancer | At the IP of a network load balancer. Extends NodePort |
| ExternalName | The Service's DNS name is an alias for an external DNS name |
| Headless | Groups Pods without a stable IP; used by StatefulSets |
Horizontal Pod Autoscaler
It increases or decreases a workload's number of Pods. It doesn't change the number of nodes: that is the cluster autoscaler's job.
Resource usage
CPU or memory, as a raw value or as a percentage of what the Pod requests.
Custom metrics
Metrics from cluster objects, such as requests per second.
External metrics
From outside the cluster, such as a Pub/Sub queue size.
- To use CPU or memory percentages, every container must set requests for that resource; otherwise the HPA takes no action on that metric.
- With several metrics, it evaluates each separately and computes a scale for each one.
- Google advises against using the HPA and the standard VPA together on CPU or memory.
kubectl autoscale deployment nginx --cpu-percent=50 --min=1 --max=10 kubectl get hpa kubectl scale deployment nginx --replicas=3 # manual scaling
Vertical Pod Autoscaler
It analyzes historical usage and adjusts containers' CPU and memory requests and limits. It works best with long-running homogeneous workloads; for sudden spikes Google recommends the HPA. It is enabled by default in Autopilot clusters.
| Mode (updateMode) | Behavior |
|---|---|
| Auto / Recreate (default) | Evicts and recreates the Pod to apply new requests. Limit it with a Pod disruption budget |
| Initial | Sets requests only at Pod creation; never changes them later |
| Off | Applies no changes: only shows recommendations to apply manually |
| InPlaceOrRecreate (preview) | Tries to change resources without recreating the Pod; recreates it if it can't |
GKE's implementation accounts for maximum node size and quotas, notifies the cluster autoscaler, and uses historical data from before it was enabled.
Resource requests in Autopilot
In Autopilot you pay for what Pods request, so requests matter. Autopilot uses them to configure nodes and validates them:
- If you don't set requests, it applies defaults: on the General-purpose, Balanced, and Scale-Out classes, 0.5 vCPU and 2 GiB of memory; 1 GiB of ephemeral storage for regular containers. DaemonSet containers get 50 mCPU, 100 MiB of memory, and 100 MiB of ephemeral storage.
- If requests are below the minimum, Autopilot raises them automatically.
- If they are above the maximum, it rejects the workload with an error.
- Minimums and maximums depend on the compute class or hardware (for example, GPUs). If you set only CPU or only memory, it fills in the other based on the allowed ratio.
- Google recommends setting each container's requests explicitly, because defaults may not be enough.
Updates and traffic splitting
Deployment rolling update
- The default strategy: replaces Pods gradually.
maxUnavailableandmaxSurge(25% by default) control how many Pods can be missing or extra during the rollout.kubectl rollout undo deployment/NAMEreturns to the previous revision (or another with--to-revision).
Gateway API
- Splits traffic between Services by weight in an HTTPRoute (for example, 90% to v1 and 10% to v2).
- It also routes by header: requests with
env: canarycan go to another version. - In Ingress these capabilities were only possible through custom annotations.
Exam scenarios
| Situation | Answer | Why |
|---|---|---|
| One workload's Pods need lots of memory and the rest don't | A separate node pool with nodeSelector | All nodes in a pool are identical. |
| Pods are pending because they don't fit on the nodes | Cluster autoscaler | Adds nodes based on Pod requests. |
| Scale replicas on pending Pub/Sub messages | HPA with an external metric | The HPA supports metrics from outside the cluster. |
| You don't know what CPU and memory to request for a stable service | VPA in Off mode | Provides recommendations without applying them. |
| A database with stable identity and one disk per replica | StatefulSet with a headless Service | Persistent identity and per-Pod volumes. |
| Nodes can't pull images from Artifact Registry | Grant Artifact Registry Reader to the nodes' service account | It is the identity they use to read. |
| Send 10% of traffic to the new version | Gateway API with weights in an HTTPRoute | Weight-based traffic splitting. |
| A new Deployment version fails | kubectl rollout undo | Returns to the previous revision. |
Common exam traps
❓ Does the HPA add nodes to the cluster?
No. It only changes the number of Pods; nodes are added by the cluster autoscaler.
❓ Does the cluster autoscaler look at actual CPU usage?
No. It decides based on Pods' resource requests.
❓ Can you configure a single node in a pool?
No. Changes affect every node in the pool.
❓ What does Autopilot do if a Pod requests more than the maximum?
It rejects the workload. If it requests less than the minimum, it raises the requests automatically.
❓ Which VPA mode only provides recommendations?
Off. Auto/Recreate is the default and recreates Pods.
❓ Why doesn't a CPU-percentage HPA scale?
Probably because some container has no CPU request; without it the percentage can't be calculated.
Did you get this topic?
Test what you just learned
In a GKE Standard cluster, the HPA creates more replicas during a spike, but the new Pods stay Pending because they don't fit on the nodes. What should you do?