A Fix for Softmax and Cross-Entropy

发生了什么

Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse , rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation , occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student…

摘要按规则整理自下方来源原文

来源