A Fix for Softmax and Cross-Entropy

What happened

Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse , rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation , occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student…

Summary assembled by rule from the sources below

Sources