The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling

Technical Summary: The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling

Problem Statement
Temperature scaling is a ubiquitous method in deep learning used to control the uncertainty of probabilistic models, serving two primary roles: calibrating the confidence of classifiers and tuning the stochasticity (often termed "diversity") of Large Language Models (LLMs). Despite its widespread adoption in industrial applications and major LLM technical reports, a rigorous theoretical analysis of its fundamental properties has remained elusive. While the method is simple—scaling model logits by a single scalar parameter $\beta$—its theoretical behavior, particularly regarding uncertainty quantification and its relationship to other calibration methods, lacks comprehensive elementary investigation.

Methodology
The authors conduct a theoretical investigation of temperature scaling, analyzing its behavior in both classification and autoregressive language modeling contexts. The methodology relies on:

1. Statistical Physics Analogies: Interpreting the tempered distribution as a Boltzmann-Gibbs distribution with an inverse temperature $\beta$, utilizing properties of the log-partition function (free energy) to derive derivatives of entropy and likelihood.
2. Information Geometry: Characterizing temperature scaling as an information projection onto the set of distributions with a fixed entropy, measured via Kullback-Leibler (KL) divergence.
3. Majorization Theory: Using the concept of majorization to formally define and compare the dispersion of probability distributions under varying temperatures.
4. Linear Algebra and Optimization: Analyzing the structural properties of linear scalers to determine necessary and sufficient conditions for preserving the ordering of class probabilities (hard predictions).
5. Counter-Example Construction: Constructing a toy autoregressive model to demonstrate non-monotonic behavior in entropy when applying "myopic" temperature scaling (independent scaling at each decoding step), which is the standard practice for LLMs.

Key Contributions and Results

• Uncertainty and Entropy in Classification:
  The paper confirms that for classification tasks, increasing the temperature (decreasing $\beta$) monotonically increases model uncertainty. This is proven rigorously using majorization theory: the tempered distribution $\pi_\beta$ majorizes $\pi_{\beta'}$ for $\beta > \beta'$, implying that the tempered distribution is more dispersed. Consequently, the Shannon-Gibbs entropy and Renyi entropies are strictly decreasing functions of the inverse temperature $\beta$. The authors also show that the derivative of the cross-entropy loss with respect to $\beta$ relates to the variance of the logits, providing a moment-matching interpretation for maximum likelihood estimation.

• The "Myopic" Paradox in LLMs:
  The authors challenge the common assumption that increasing temperature always increases diversity in LLMs. While standard temperature scaling on a joint distribution (which is often intractable) behaves monotonically, practical LLMs use "myopic" temperature scaling, where $\beta$ is applied independently at each decoding step. The paper demonstrates via a counter-example that for autoregressive models, the joint entropy of the myopic tempered distribution is not necessarily monotonic with respect to $\beta$. Specifically, decreasing $\beta$ (increasing temperature) can initially decrease the joint entropy before eventually increasing it, due to the interplay between early-token certainty and conditional entropy of subsequent tokens.

• Geometric Characterization (Information Projection):
  The paper provides a geometric interpretation of temperature scaling: a tempered model is the unique information projection (closest in KL divergence) of the original model onto the set of distributions possessing a specific target entropy. This establishes temperature scaling as the optimal method for adjusting uncertainty to a specific level without altering the underlying energy landscape more than necessary.

• Uniqueness as an Accuracy-Preserving Linear Scaler:
  The authors prove a defining structural property: temperature scaling is the only linear scaler (acting on logits or log-probabilities) that preserves the hard predictions (the $\arg\max$) of the original model. Any linear transformation that preserves the ordering of class probabilities must be a scalar multiple of the identity matrix plus a constant vector shift, which reduces mathematically to temperature scaling. This implies that any attempt to create a more flexible, accuracy-preserving calibration method must necessarily depart from linearity.

Significance and Claims
The paper positions itself as a timely, elementary, yet rigorous inspection of a widely used tool. Its significance lies in:

1. Clarifying Limits: It establishes that while temperature scaling is a principled uncertainty control mechanism for classifiers (monotonically increasing uncertainty via majorization), these guarantees do not automatically extend to the sequence-level diversity of autoregressive language models, where non-monotonic effects can occur.
2. Structural Justification: It provides a theoretical justification for the empirical success of temperature scaling over matrix scaling in high-accuracy regimes. Since temperature scaling is the unique linear method that preserves accuracy, it exhausts the degrees of freedom available to linear recalibrators once accuracy is fixed.
3. Practical Implications: The authors suggest that the counter-intuitive behavior of temperature in LLMs may explain why other stochasticity control methods (like nucleus sampling) are sometimes preferred and why empirical findings show weak correlations between temperature and text creativity. They also advise that for calibrating highly accurate classifiers, temperature scaling is theoretically superior to matrix scaling or Dirichlet calibration due to its accuracy-preserving nature.

The work concludes by noting that while the theoretical properties are now better understood, the question of why such a simple linear model performs so well in practice (given that optimal scalers are often nonlinear) remains an open problem for future research.