project

NVIDIA Resiliency Extension

A pip-installable Python library from NVIDIA that adds fault-tolerance primitives to PyTorch training: asynchronous checkpointing, in-process restart, and a heartbeat-monitoring launcher for in-job restart.

Known aliases

nvidia-resiliency-ext
NVRx

Relationships

No evidence-backed relationships are recorded.

Current clusters

build1 publisher

Blocking checkpoint writes took up to 40 percent of wall time on AWS's H100 test runs

AWS's NVRx walkthrough puts synchronous checkpointing ahead of GPU faults as the source of idle time on its FSDP jobs, and pairs a background save with a restart that re-enters training without cycling the container.

Publishers:aws.amazon.com

Reality

Evidence54
Adoption20
Hype gap+18
Incentives76
Confidence56