viable/strict/1790930233: [FSDP2] Defer gradient upcasts to the reduce-scatter copy-in (#198668)
- PyTorch: 1792 events in the last 90 days
- PyTorch: 1770th Release in the last 90 days
- Previous: earlier the same day · trunk/89492c102edbb5378e03b660810a6640a4c9d462
What happened
Authored with Claude Code. #194434 sets the unsharded parameter's grad_dtype to its unsharded gradient dtype. With FP32 parameters and param_dtype=torch.bfloat16, that dtype is FP32 whether reduce_dtype is None or torch.float32 (torchtitan's default). Autograd therefore casts every BF16 gradient to FP32 on the compute stream, one cast per parameter, and FSDP holds FP32 unsharded gradients until the reduce-scatter co…
Summary assembled by rule from the sources below