Start with the arithmetic that makes checkpoint frequency the dominant lever. Databricks states that a job checkpointing once a day loses, on average, 12 hours of duplicate work per failure [5]. That is the expected value of a uniform failure arrival inside a 24-hour interval: half the interval. Cut the interval to 30 minutes and the same model gives 15 minutes of rework [11]. The GPU itself hasn't changed at all; what changed is the write path.
So what stops teams from checkpointing every 30 minutes is the cost of the write itself. The default is `torch.save` on rank 0 [6]. It serialises, it blocks, and the accelerators sit there. PyTorch's distributed checkpoint inverts that: every rank writes its own shard in parallel, alongside a small `.metadata` file describing how the shards compose back into full tensors [7]. Save time then falls roughly as 1/N in the number of ranks [8].
The `.metadata` file is the part worth reading closely, because it is what makes recovery elastic. Since it records the global layout, the same checkpoint reloads onto a different GPU count, with DCP re-planning which bytes each new rank needs [9]. That is the difference between a node failure meaning "wait for replacement capacity" and meaning "resume smaller". Databricks also pushes back on the assumption that DCP is only for sharded models, noting that it shards and parallel-writes state even for DDP jobs where every rank holds an identical replica [10].
Parallel writes still leave a synchronous barrier: the loop waits until bytes are durable, which for a large checkpoint to a remote volume is tens of seconds of idle accelerators [12]. `async_save` splits it into a fast staging copy plus a background upload that overlaps continued training, so the loop pays for the copy and not the upload [13]. On AI Runtime, `UCVolumeWriter` and `UCVolumeReader` implement DCP against Unity Catalog volumes, staging I/O through local NVMe and marking a checkpoint complete only after its data has fully landed [14].
The failure-rate numbers are where I would want my own measurements before adopting the frame. The back-of-the-envelope model assumes a 1% annualised failure rate per GPU, giving a 256-GPU job over 30 days about a 19% chance of an interruption and 57% at 1,024 GPUs [3]. Databricks then grounds it against the 608-GPU H100 Delta supercomputer, which saw failures every 1.9 hours, implying roughly 36 hours mean time to failure for a 32-GPU job [4]. Those two views are not close. 1.9 hours across 608 GPUs is about 1,155 GPU-hours between failures; a 1% annualised rate is roughly 876,000 GPU-hours per failure, which is two and a half orders of magnitude apart [15]. The stated caveat is that the model covers infrastructure-level issues only [3]. Whichever number transfers to your fleet depends on your hardware age, your thermals, and how much of the tail your health checks catch, and Databricks routes that question to a separate post on detecting and isolating unhealthy GPUs [16].
The loader half of the argument gets less mechanism in this post, but the claim is sharp: a pipeline that cannot keep pace starves the GPUs and erodes goodput just as surely as a crash [2]. In one respect that failure mode is quieter than a crash, since a crash at least shows up in your logs.
