Blocking checkpoint writes took up to 40 percent of wall time on AWS's H100 test runs

The save path is two calls. `async_save()` hands the state dict to a background process and returns immediately, and a matching `finalize_async_save()` runs before the next save to commit the previous write ^[5]. So the checkpoint interval has to be longer than the time it takes to flush a shard. If it is not, the loop stops at finalize instead of at save.

Paired with FSDP `LOCAL_STATE_DICT`, each rank writes its own shard directly, with no all-gather and no rank-0 bottleneck ^[6]. The cost moves off the collective and onto concurrent writes against shared storage. The post persists checkpoints to Amazon FSx ^[14].

AWS reports that synchronous saves consumed up to 40% of total wall time at the cluster sizes in the post ^[2]. Take out a phase that occupies 40% of the clock and 60% remains, so the ceiling on the speedup is 1/0.6, about 1.67x for the same work ^[18]. Three things have to match for that 40% to describe another job: bytes of state per rank, the step time available to hide the write behind, and the write bandwidth the storage tier delivers at that rank count. The runs were 2 to 8 nodes of p5.48xlarge, 8 NVIDIA H100 80 GB GPUs each, so 16 to 64 GPUs ^[12][13][17]. On a job that saves once an hour, a 40% share means each save takes 24 minutes ^[19].

The failure mode on the other side is the cascade: NCCL timeouts propagate from one faulted GPU to healthy workers, pods crash and restart out of sync, and the cluster burns GPU hours while making zero progress ^[3]. `inprocess.Wrapper` wraps the train function. On an unhandled exception or an NCCL hang, NVRx aborts the active process group, health-checks each rank's GPU, NVLink and NIC, re-rendezvouses the survivors, and re-enters the wrapped function from the latest checkpoint ^[7]. The interpreter, the CUDA allocator and outer-scope objects stay alive ^[8]. AWS describes this as recovering in seconds without touching the container lifecycle ^[9]. It needs survivors to rendezvous with, and node loss falls to the cluster orchestrator, the third layer ^[11].

Hard faults go to `ft_launcher`: SIGKILL, OOM kills, OS-level hangs ^[10]. Each rank runs a RankMonitorClient, and the launcher compares heartbeats against timeouts set explicitly on the command line ^[10]. Those timeouts are the tuning surface. Set them tight and a slow step is scored as a dead rank; set them loose and a hung job keeps billing until they expire.

Adoption cost, by the post's own account, is a pip install of nvidia-resiliency-ext plus ordinary imports, with no PyTorch fork, no custom kernels and no recompile, and each primitive can be taken on its own ^[4]. The Kubernetes side is the part that is not a pip install: GPUs and EFA adapters surfaced as extended resources by two device plugins, headless Services so workers find peers over DNS instead of hardcoded IPs, node affinity and tolerations to land a full 8-GPU allocation per node ^[16][15]. AWS says the code is published to reproduce the runs ^[13].