ciflow/trunk/198302: [test] Reuse process groups across FSDP hybrid-shard subtests
- PyTorch: 1354 events in the last 90 days
- PyTorch: 1337th Release in the last 90 days
- Previous: earlier the same day · ciflow/trunk/198301: [test] Reuse device meshes across FSDP2 ND-mesh subtests
What happened
TestFSDPHybridShard is a continuous test, so its 8 worker processes are reused for every test in the class. test_fsdp_hybrid_shard_basic_setup built a new device mesh in each subtest and test_fsdp_hybrid_shard_parity built new HSDP process groups in each subtest. On RCCL every communicator keeps a ~40 MB proxy-ops segment per rank in /dev/shm until the workers exit. basic_setup left ~7 GB behind and parity then hit…
Summary assembled by rule from the sources below