framework

Fully Sharded Data Parallel

PyTorch's training strategy that shards model parameters, gradients and optimizer state across ranks to fit larger models in aggregate GPU memory.

Current clusters

build1 publisher

Blocking checkpoint writes took up to 40 percent of wall time on AWS's H100 test runs

AWS's NVRx walkthrough puts synchronous checkpointing ahead of GPU faults as the source of idle time on its FSDP jobs, and pairs a background save with a restart that re-enters training without cycling the container.

Publishers:aws.amazon.com

Reality

Evidence54
Adoption20
Hype gap+18
Incentives76
Confidence56