← Back to events
ActiveAIRelease

ciflow/trunk/198301: [test] Reuse device meshes across FSDP2 ND-mesh subtests

What happened

test_2d_mlp_with_nd_mesh and test_3d_mlp_with_nd_mesh build a global mesh and pass it to their subtest helpers, but each helper overwrote it with a freshly built mesh, so every subtest created new NCCL communicators. test_shard_placement_fn_tp_ep rebuilt its meshes in every subtest even though they depend only on tp_degree and dp_replicate. On RCCL every communicator keeps a ~40 MB proxy-ops segment per rank in /dev…

Summary assembled by rule from the sources below

Why it's spreading

Sources