Gaussian equivalence and attention spectra
Gaussian equivalence for single- and multi-head self-attention, using random matrix theory and free probability to analyze attention spectra.
This research theme studies the spectra of self-attention matrices through Gaussian equivalence, random matrix theory, and free probability.
Gaussian Equivalence for Self-Attention establishes the spectral analysis of the attention matrix. Gaussian Equivalence for Multi-Head Self-Attention develops the multi-head setting, including value and output projections and dependencies across heads.
The Japanese Neural Network Society commentary introduces the random-matrix background and attention spectra. The spring JPS, AISTATS, Komaba, and NCPRT talks present this research; their records and available materials are listed below.
Random matrix spectra and asymptotic freeness provide the mathematical tools. Inverse temperature and softmax concentration examines how attention weights concentrate as the score scale and context length change. The relation graph connects these two views of self-attention.
Findings (7)
Papers (3)
Talks (4)
Notes (1)
Connections
This topic …
uses
- Random matrix spectra · uses spectral analysis
Gaussian equivalence gives a tractable random-matrix surrogate for analyzing the limiting singular-value distribution of attention matrices.
- Asymptotic freeness · uses asymptotic freeness
Asymptotic freeness provides free-probability tools for combining large random matrices in the spectral analysis of attention.
extends
- Dynamical isometry · extends spectral analysis
The singular-value viewpoint used to study signal propagation and dynamical isometry also motivates spectral analysis of self-attention layers.
Referenced by
- Inverse temperature and softmax concentration · connects concentration and spectra
Both themes study self-attention. Inverse-temperature scaling describes concentration within softmax rows; Gaussian equivalence describes spectral distributions of attention matrices, including the multi-head setting.