Fully Sharded Data Parallel

Databricks puts your training bill in the data loader and the checkpointer

The argument is that GPU failures are the expected case at scale, so recovery speed sets goodput and therefore spend. Two subsystems most teams write once and forget decide how much idle time each failure costs.