viable/strict/1791021071: [fsdp2][mp] per-param mixed_precision policy (#196626)
- PyTorch: 1850 events in the last 90 days
- PyTorch: 1826th Release in the last 90 days
- Previous: earlier the same day · trunk/47df693ee1bd31c15dc89fbc4884c99ddd82a0a4
What happened
Summary: We want to enable per-param mixed precision policy in FSDP2 for several reasons including KDA, etc. Key Points All-gather: During all-gather, existing homogenous path remains unchanged. When parameters require different compute dtypes, FSDP groups staging copies by source/target dtype and invokes the existing uint8 communication buffer, keeping only one AG per fully_shard. As seen in benchmarking results be…
Summary assembled by rule from the sources below