Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models

Technical Summary: Respect Your Zero-Shot Uncertainty

Problem Statement
Test-time adaptation (TTA) enables vision-language models (VLMs) to improve recognition accuracy under distribution shifts using unlabeled target data. However, existing TTA methods often degrade model calibration, rendering predictive confidence unreliable for downstream decision-making. The paper identifies a specific failure mode termed prediction-preserving sharpening: TTA can significantly increase predictive confidence and reduce entropy even when the top-1 prediction (and thus its correctness) remains unchanged compared to the zero-shot baseline.

The authors argue that existing label-free calibration approaches are insufficient. Prompt-coupled methods are limited to specific adaptation mechanisms, while plug-in methods like SaLS rely on logit-range statistics. The paper demonstrates that logit range (the difference between maximum and minimum logits) fails to capture the distribution of probability mass among remaining classes and does not correlate strongly with calibration error. Consequently, there is a need for a label-free, sample-specific signal that captures changes in the complete predictive distribution.

Methodology: Zero-Shot-Anchored Entropy Calibration (ZAEC)
To address this, the authors propose ZAEC, a label-free, post-hoc calibration method that uses the zero-shot entropy of a sample as a conservative, sample-specific uncertainty reference.

1. Signal Identification: The method identifies samples where the adapted predictive distribution is "sharpened" relative to the zero-shot baseline. Specifically, it targets the subset of samples $I^-$ where the adapted entropy $H(p^{TTA}_i)$ is strictly less than the zero-shot entropy $H(p^{ZS}_i)$.
2. Selective Correction: For samples in $I^-$, ZAEC applies a minimal temperature scaling factor $T^*_i \geq 1$ to the adapted logits. The temperature is chosen such that the resulting entropy exactly matches the original zero-shot entropy: $H(\text{softmax}(z^{TTA}_i / T^*_i)) = H(p^{ZS}_i)$.
3. Preservation: For samples where the entropy is not reduced (i.e., $H(p^{TTA}_i) \geq H(p^{ZS}_i)$), no correction is applied ($T^*_i = 1$).
4. Properties: The method relies on the monotonic relationship between temperature and entropy. Crucially, positive temperature scaling preserves the complete class ranking and the top-1 prediction. Therefore, ZAEC corrects confidence without altering classification accuracy or class ordering. It requires no labeled calibration data and introduces no learnable parameters.

Key Contributions

• Identification of Failure Mode: The paper reveals "prediction-preserving sharpening" as a primary source of calibration degradation in TTA, showing that confidence gains often outpace accuracy gains on samples where the prediction class remains unchanged.
• Correlation Analysis: The authors demonstrate a strong negative correlation between zero-shot-relative entropy reduction and Expected Calibration Error (ECE) across diverse TTA methods and datasets. They show that logit-range alignment is insufficient to characterize calibration quality.
• ZAEC Algorithm: They introduce a mechanism-agnostic, post-hoc method that selectively restores zero-shot entropy via minimal temperature scaling. The method is proven to preserve class rankings and classification accuracy.
• Empirical Performance: Extensive experiments show that ZAEC consistently achieves the lowest method-level average ECE among evaluated post-hoc methods.

Experimental Results
The authors evaluated ZAEC across five representative TTA methods (TPT, TPS, TDA, DPE, DLAE) and 15 datasets (including ImageNet variants and fine-grained classification datasets) using CLIP backbones (ViT-B/16 and RN50).

• ECE Reduction: On ViT-B/16, raw TTA methods exhibited macro-average ECEs ranging from 7.47% to 28.47%. ZAEC reduced these values to a range of 3.84% to 8.19%, achieving the lowest average ECE for every TTA method and benchmark group compared against raw TTA, Temperature Scaling, and SaLS.
• Consistency: Similar trends were observed on the RN50 backbone, confirming the method's robustness across different model architectures.
• Ablation Studies:
  • Correcting only entropy-reduced samples (ZAEC) yielded better results than correcting all samples or only entropy-increased samples.
  • Full restoration of zero-shot entropy ($\lambda=1$) yielded lower ECE than partial restoration.
  • Boundary Case: The authors note a limitation on the Caltech101 dataset. In cases where the zero-shot model is underconfident and TTA correctly sharpens the distribution to match empirical accuracy, ZAEC's conservative restoration of zero-shot entropy can reintroduce underconfidence, slightly increasing ECE.

Significance and Claims
The paper claims that effective VLM adaptation must improve recognition performance without sacrificing the reliability of predictive confidence. By anchoring calibration to the zero-shot uncertainty, ZAEC provides a conservative, label-free solution to the overconfidence induced by TTA. The authors emphasize that their method preserves the utility of the adapted model (accuracy and rankings) while mitigating the risk of high-confidence errors. They conclude that future work should explore adaptive anchors for scenarios where zero-shot underconfidence makes the current conservative approach suboptimal.