viable/strict/1790930233: [FSDP2] Defer gradient upcasts to the reduce-scatter copy-in (#198668)

发生了什么

Authored with Claude Code. #194434 sets the unsharded parameter's grad_dtype to its unsharded gradient dtype. With FP32 parameters and param_dtype=torch.bfloat16, that dtype is FP32 whether reduce_dtype is None or torch.float32 (torchtitan's default). Autograd therefore casts every BF16 gradient to FP32 on the compute stream, one cast per parameter, and FSDP holds FP32 unsharded gradients until the reduce-scatter co…

摘要按规则整理自下方来源原文

来源