Skip to content

Build1 publisher3 min readPublished

GKE Pod snapshots skip large-model loading as long as the node's driver and kernel match

Google says GKE Pod snapshots, which restore saved CPU and GPU memory, cut startup latency by up to 89% and load a 70B model in 37 seconds. When a snapshot stops matching its node, the Pod starts cold with no error, so teams must keep snapshots valid across upgrades.

The Engineer · Build desk

Illustration accompanying GKE Pod snapshots skip large-model loading as long as the node's driver and kernel match

What happened

  • A snapshot holds open file descriptors, threads, CPU registers, memory, the root filesystem, EmptyDir volumes and tmpfs mounts, so a new replica never runs model initialization.
  • Pods must run in GKE Sandbox under gVisor; Autopilot clusters already have it, while Standard clusters need a node pool with gVisor turned on.
  • Two custom resources configure the feature: PodSnapshotStorageConfig points at the bucket, and PodSnapshotPolicy selects Pods by label and sets the trigger and retention.
  • Codeway now starts H100 instances for a specific job and shuts them down when the job finishes.
  • Whole-pod snapshots do not work on E2 machine types, multi-GPU Pods are supported only on L4 GPUs, and Multi-Instance GPU sharing is not supported.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Standard-cluster teams have to move model-serving Pods onto a gVisor node pool to use snapshots, so adopting the feature also means running those workloads in the sandbox runtime.
  • constraint Any node pool upgrade that changes the gVisor kernel or GPU driver can void existing snapshots, so upgrades and fresh snapshot capture have to be planned as one change.
  • cost Code written for a fresh process start has to be changed to re-create keys and certificates, re-read its environment and reconnect after restore, or it keeps running on stale state.
  • exposure The snapshot bucket holds process memory, including key material present at freeze time, so in my view its access controls need to match those of a secrets store.

The restore path has a strict gate. GKE hashes the Pod's essential runtime fields, which it calls the distilled Pod spec, and embeds the hash in the snapshot. A Pod restoring from it must produce the same hash [11]. The target node must also match on machine series and CPU architecture, N2 to N2 or G2 to G2, and on the gVisor kernel and GPU driver versions captured at snapshot time [12]. Where no compatible snapshot exists, the Pod starts normally [13].

Failing open is the right default for a serving fleet. A slow start beats a bad restore. The cost is visibility. InfoQ's analysis is that a node pool upgrade may change the gVisor kernel or GPU driver version, after which existing snapshots stop matching and Pods start cold with no error [14]. The fallback is polite enough that nobody gets paged. Mohana Narasimha G., a senior DevOps and MLOps engineer, wrote in reply to a LinkedIn analysis of the release: "The restore path is compelling, but I suspect snapshot invalidation will be the harder platform problem than capture itself." [10] I would want a count of restored starts against normal starts on every rollout, because slower startup is the only symptom of a lost snapshot [14].

The application has to cooperate too. A restored process resumes holding whatever it held when frozen, so encryption keys and certificates created before the snapshot must be re-created [16]. Environment variables sit in application memory, where gVisor cannot reliably find and replace them. A workload that needs new values reads them from /proc/gvisor/spec_environ [17]. External connections are terminated on restore, persistent volumes are not checkpointed, and user-added iptables or nftables rules and routes are not restored [18]. Whole-pod snapshots include process memory and are stored in Cloud Storage, so any key material the process held at freeze time is in the bucket [3].

Google's benchmarks follow general availability in May, on clusters at version 1.35.3-gke.1234000 or later [3]. Restore is staged. The gVisor kernel comes back first, usually within a few seconds, and the application starts running while its memory is still loading in the background [19]. Whether Google's 70B model is fully resident at 37 seconds depends on where the timer stopped [1][19]. The InfoQ report does not give the baseline load times behind the 89% figure. For the result to carry over to another fleet, the workload has to land on matching hardware and versions, and its first requests have to tolerate memory that is still arriving [12][19].

Codeway's numbers are easier to interpret. Its Retake platform already had a custom caching layer for compiled artifacts that brought startup down to a minute [8]. Ahmet Furkan Çomak, Codeway's lead DevOps engineer, said Pod snapshots cut that to "just 8 seconds" [8]. The drop is about 87%, measured against a baseline that was already optimized [1].

The rootfs-only scope loosens the gate. GKE skips the hash comparison, and because process memory is not restored, those snapshots can cross machine families, including to E2 [15]. Without process memory, the initialization that loads the model has to run again [2].

What to watch

  • Whether GKE exposes a metric or event when a Pod falls back to a normal start after failing the compatibility check.
  • Whether Google publishes baseline load times and the exact measurement point behind the 37-second and 15-second figures.
  • Whether multi-GPU snapshot support extends beyond L4 GPUs or Multi-Instance GPU sharing becomes supported.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories