viable/strict/1791021071: [fsdp2][mp] per-param mixed_precision policy (#196626)

发生了什么

Summary: We want to enable per-param mixed precision policy in FSDP2 for several reasons including KDA, etc. Key Points All-gather: During all-gather, existing homogenous path remains unchanged. When parameters require different compute dtypes, FSDP groups staging copies by source/target dtype and invokes the existing uint8 communication buffer, keeping only one AG per fully_shard. As seen in benchmarking results be…

摘要按规则整理自下方来源原文

来源