Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models

Technical Summary: Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models

Problem Statement
Recent research has focused heavily on calibrating Large Language Models (LLMs) to ensure their uncertainty estimates accurately predict the likelihood of generating a valid answer. However, a critical gap exists: existing work rarely evaluates whether model uncertainty aligns with human uncertainty. While a measure may be well-calibrated to model accuracy, its values may not correspond to human intuition, making them difficult for users to interpret. Furthermore, most uncertainty quantification (UQ) methods require auxiliary generations (e.g., Monte Carlo dropout), rendering them unsuitable for real-time model control or user feedback. This paper addresses the need for inference-time UQ measures that are simultaneously calibrated (predicting model correctness) and aligned (correlating with human group-level uncertainty).

Methodology
The authors evaluate a collection of inference-time uncertainty measures across multiple LLMs (LLaMa-3.2 1B/3B, LLaMa-3.1 8B, and Mistral 7B 0.1/0.3, both base and instruction-tuned variants). The study utilizes two human survey datasets:

1. Pew Dataset: 38 questions from Pew Research surveys (used as a baseline).
2. Roper Dataset: 2,998 questions sampled from the Roper Center for Public Opinion Research (2017–2023).

Human Uncertainty Definition: Human uncertainty is approximated by the level of disagreement among groups of survey respondents. It is calculated as the entropy of the normalized response percentages for each question.

Uncertainty Measures Evaluated:
The study focuses on measures calculable during a single generation pass without auxiliary sampling:

• Top-1 Probability: The probability of the most likely token.
• Entropy-based Measures:
  • Total Entropy: Shannon entropy over the full vocabulary distribution.
  • Top-k Entropy: Entropy calculated over the top $k$ tokens ($k \in \{5, 10, 25, 50, 100\}$), normalized to maintain relative ratios.
  • Choice Entropy: Entropy calculated specifically over the normalized probabilities of the provided answer choices.
  • Top-p Entropy: Entropy over the tokens selected via nucleus sampling (nucleus size $p \in \{0.5, 0.75, 0.9, 0.95\}$).
• Top-p Size: The number of tokens in the nucleus set.

Evaluation Metrics:

1. Alignment:
   • Overt Agreement: The frequency with which the model's top-choice matches the human group's top-choice.
   • Preference Ordering: Measured via normalized Kendall $\tau$ distance between model and human preference rankings.
   • Uncertainty Correlation: Pearson correlation between model uncertainty scores and human group uncertainty (entropy of responses) across questions.
2. Calibration:
   • Correctness Correlation: Spearman correlation between uncertainty measures and binary correctness on the MMLU benchmark.
   • Distributional Calibration (JSD Shift): A novel metric measuring the shift in Jensen-Shannon Distance (JSD). The authors hypothesize that a well-calibrated measure will show a significant increase in the distance between the model's answer distribution and the correct answer distribution when moving from high-certainty to low-certainty questions.

Key Results

• Alignment vs. Preference: Models showed moderate agreement with humans on the top answer (beating random chance), but showed no significant alignment in overall preference ordering (Kendall $\tau$ distances were effectively random).
• Uncertainty Alignment: Despite the lack of preference ordering alignment, several measures showed strong correlation with human group uncertainty.
  • Top-performing measures: Choice entropy, Total entropy, all Top-k entropies, and Top-p entropy (for $p \ge 0.9$) consistently showed significant correlation ($|r| \ge 0.3$) across all models.
  • Top-p Size: Showed weak but significant correlation for $p \le 0.9$, contradicting some prior assumptions.
  • Top-1 Probability: Did not show consistent significant correlation.
• Calibration: The measures that aligned well with human uncertainty also demonstrated moderate to strong evidence of calibration on MMLU.
  • Choice Entropy emerged as the most robust measure, showing negative correlation with correctness (lower uncertainty = higher accuracy) across nearly all models and subjects.
  • JSD Shift: Permutation tests confirmed that for most measures, the shift in distributional distance between high and low certainty states was statistically significant, indicating the measures effectively distinguish between certain and uncertain model states.

Key Contributions

1. Top-p and Bayesian Connection: The authors note that top-p selection in LLM decoding is functionally equivalent to the Bayesian highest density credible set, a connection previously unnoted in the UQ literature.
2. Decoupling Alignment and Preference: The work observes that entropy-based inference-time measures can align with human uncertainty even when models and humans disagree on specific answer preferences or ordering.
3. Novel Calibration Metric: The development of a ground-truth distributional calibration measure based on the shift in the Jensen-Shannon distance metric to directly evaluate the impact of certainty on answer distributions.
4. Empirical Evidence: The study provides evidence that aligned inference-time measures are also calibrated in terms of both correctness correlation and distributional analysis.

Significance and Claims
The paper claims to identify a set of inference-time UQ measures that are effective and more intuitively interpretable by human users because they align with human uncertainty behavior. This is significant because it suggests that models can provide real-time confidence signals that users can understand, even if the models do not perfectly mimic human answer selection logic.

The authors modestly conclude that while these measures show strong initial evidence of alignment and calibration, they are not perfect. They emphasize that future work is needed to:

• Find measures that are highly calibrated without sacrificing alignment.
• Extend these findings to open-ended generation contexts (proposing a framework to convert open-ended responses into multiple-choice evaluations).
• Investigate individual-level uncertainty alignment rather than just group-level.
• Test these measures in actual human-LLM cooperative applications to determine if they improve user trust and task efficacy.

The study is limited to multiple-choice contexts and models with 8 billion parameters or fewer, noting that results may vary in larger models or open-ended scenarios.