Gaussian Equivalence for Multi-Head Self-Attention

Abstract

Gaussian equivalence for multi-head self-attention preserves the limiting spectral law of the centered output when softmax attention is replaced by rescaled scores plus Gaussian noise. The analysis includes key-dependent value and output projections and separates the effects of head allocation, projection widths, across-head sharing, and within-head key–value dependence.