trunk/2f2f675bc57fb8bde0bf9c21ed41dd8f44de6a45: [c10d] Cut ProcessGroupNCCLErrorsTest runtime by ~13 min (#199826)
- PyTorch: 2035 events in the last 90 days
- PyTorch: 2008th Release in the last 90 days
- Previous: earlier the same day · trunk/93f87db32e92102353721df25a2413e2b99d0b4e
What happened
ProcessGroupNCCLErrorsTest spends ~830s per CUDA distributed config in three tests: testNCCLErrorsNonBlocking (242s): the watchdog sleeps TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC * 4 (60s by default) after each of the 4 failed works. Set it to 100ms, as the other timeout tests already do. testTimeoutTearsDownWithoutRethrow{,WithCleanUp} (~290s each): the death-test child aborts with a CUDA context, and CI leaves core dum…
Summary assembled by rule from the sources below