trunk/7c647ebb2837fd3a9618babc81e3558c64fe69f3: Fix test_ddp_zero_overlap parity check on 4 or more ranks (#197890)
- PyTorch: 1319 events in the last 90 days
- PyTorch: 1304th Release in the last 90 days
- Previous: earlier the same day · trunk/618de55dfddff78055974a180bfc5b60c5c81d07
What happened
Summary: test_ddp_zero_overlap compares a DDP model whose optimizer step is overlapped via ZeroRedundancyOptimizer against a reference DDP model driven by a local SGD, and asserts the parameters match afterwards. The two models were not reducing gradients the same way. The overlapped model must run 2 warmup iterations (3 with static_graph) because the ZeRO comm hook only starts updating parameters once DDP has rebui…
Summary assembled by rule from the sources below