Fault tolerant distributed training on Amazon EKS using NVRx
- AWS: 96 events in the last 90 days
- Previous: earlier the same day · AWS reimagines the getting started experience
What happened
Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery.
Summary assembled by rule from the sources below
Why it's spreading
Sources
Official
- Fault tolerant distributed training on Amazon EKS using NVRxAWS Machine Learning