Technical Summary: Quantile-Adaptive Temperature Scaling for Confidence Calibration

1. Problem Definition and Motivation

Deep neural networks frequently produce poorly calibrated confidence estimates, often exhibiting overconfidence even when predictions are incorrect. While Temperature Scaling (TS) remains the standard post-hoc calibration method due to its simplicity, it relies on a global, uniform rescaling of logits. The authors argue that this approach fails to address the highly heterogeneous structure of miscalibration observed across the confidence spectrum.

Through quantile-wise analysis, the paper demonstrates that the largest discrepancies between correctness and confidence do not occur uniformly. Instead, the location and shape of miscalibration vary significantly depending on the regime:

• Standard Settings: Errors are concentrated in the lowest confidence quantiles.
• Long-tailed Settings: Errors shift toward medium-confidence quantiles due to the prevalence of mid-frequency classes.
• Distributional Shifts: Global rescaling methods struggle because covariate perturbations alter the absolute scale of confidence values while preserving their relative ordering.

Existing methods, including advanced variants like Group Calibration (GC) or input-dependent temperatures, implicitly assume a smooth, globally consistent transformation. This structural rigidity prevents them from effectively targeting specific regions where miscalibration is most severe, leading to suboptimal performance under challenging scenarios like class imbalance and domain shifts.

2. Methodology: Quantile-Adaptive Temperature Scaling (QaTS)

To address these limitations, the authors propose Quantile-Adaptive Temperature Scaling (QaTS), a post-hoc calibration method that adapts the temperature parameter based on a prediction's empirical confidence quantile rather than its raw confidence magnitude.

Core Mechanism

1. Quantile Reparameterization:
   The method maps raw confidences $p$ into a uniform quantile space $\tau \in [0, 1]$ using the empirical cumulative distribution function (CDF) of the calibration set. This transforms the Expected Calibration Error (ECE) objective into an integral over the uniform quantile axis, preventing dense confidence regions from dominating the error metric and making the heterogeneous structure of miscalibration explicit.

2. Sample-Wise Temperature Function:
   Instead of a single scalar $T$, QaTS defines a sample-dependent temperature $T(x)$ as a monotone linear function of the empirical quantile $q(x)$:
   $$T(x) = a \cdot (1 - q(x)) + b$$
   where $a$ and $b$ are learnable positive parameters.

   • Low-quantile predictions (typically overconfident and error-prone) receive higher temperatures, softening the confidence.
   • High-quantile predictions receive smaller adjustments, preserving accuracy and avoiding unnecessary distortion.

3. Optimization:
   The parameters $a$ and $b$ are optimized on a calibration set by minimizing the Negative Log-Likelihood (NLL), a strictly proper scoring rule. This surrogate optimization aligns the calibrated probabilities with empirical correctness frequencies, effectively minimizing the reparameterized ECE.

4. Inference:
   For a test sample $x$, logits are scaled by the computed $T(x)$ before applying the softmax. Because $T(x)$ depends on the rank (quantile) of the confidence rather than its absolute value, the method inherits the rank-based stability of quantiles. This makes QaTS robust to global logit scaling effects often associated with covariate shifts, provided the relative ordering of confidences is approximately preserved.

3. Key Contributions

The paper outlines three primary contributions:

1. Quantile-Wise Analysis: A detailed analysis revealing that calibration errors are highly heterogeneous across the confidence spectrum and vary significantly across different regimes (standard vs. long-tailed). This analysis exposes the structural limitations of existing TS-based methods that cannot target specific error-concentrated regions.
2. QaTS Algorithm: The introduction of a simple, efficient strategy that operates directly in quantile space. It formalizes a quantile reparameterization of ECE and introduces a monotone temperature function that adapts to the relative rank of each prediction.
3. Comprehensive Empirical Validation: Extensive experiments across 162 settings, including standard computer vision, medical imaging, text classification, and semantic segmentation, under various conditions (long-tailed distributions, distributional drifts).

4. Experimental Results

The authors evaluate QaTS against state-of-the-art post-hoc baselines, including Uncalibrated models, TS, Isotonic Regression (IR), AdaTS, Group Calibration (GC), and FeatClip.

• Standard Benchmarks: On CIFAR-10, CIFAR-100, and ImageNet, QaTS consistently achieves the lowest ECE, outperforming all competing methods across CNN and Vision Transformer backbones.
• Long-Tailed Distributions: In scenarios with severe class imbalance (e.g., CIFAR-100-LT, ImageNet-LT), standard methods often fail or degrade. QaTS demonstrates substantial robustness, reducing ECE by 80–90% in some cases compared to baselines like AdaTS and GC.
• Distributional Shifts: Under synthetic corruptions (CIFAR-100-C) at varying severity levels, QaTS significantly outperforms accuracy-preserving methods, achieving 2–3× lower ECE. Notably, it outperforms HFE, a method specifically designed for distributional drifts that requires access to out-of-distribution (OOD) samples for every target domain. QaTS achieves this without requiring any target domain data.
• Generalization: The method shows superior performance in medical image classification (HAM10000, MedMNIST2D), semantic segmentation (Pascal VOC), and text classification (20 Newsgroups).
• Accuracy Preservation: Unlike FeatClip, which improves calibration at the cost of significant accuracy degradation (up to 4–6%), QaTS preserves the predicted class rankings, maintaining discriminative performance while improving calibration.
• Compatibility: QaTS acts as a complementary post-hoc layer, further reducing ECE when applied on top of models trained with calibration-aware objectives (e.g., Brier loss, Focal Loss, Label Smoothing).

5. Significance and Claims

The paper claims that QaTS offers a simple, efficient, and robust foundation for post-hoc calibration. Its significance lies in:

• Addressing Heterogeneity: By moving from raw confidence scores to quantile space, QaTS explicitly targets the regions where miscalibration is most severe, overcoming the "one-size-fits-all" limitation of global temperature scaling.
• Robustness to Shift: The rank-based formulation ensures stability under distributional shifts where absolute confidence magnitudes change but relative ordering is preserved.
• Practicality: It requires no retraining, no access to model weights or gradients, and no auxiliary networks, making it scalable to large pretrained models and frozen backbones.
• Universal Applicability: The method demonstrates consistent effectiveness across diverse tasks, architectures, and challenging real-world scenarios, positioning it as a universal calibration strategy that improves reliability without compromising predictive accuracy.