Technical Summary: Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers

Problem Statement

Confidence calibration is critical for the reliability of Large Language Models (LLMs), ensuring that a model's expressed confidence aligns with its actual predictive performance. While existing training-free calibration methods have been extensively studied for single-answer question answering (QA), this paper identifies a critical failure mode when applied to questions with multiple valid answers.

The core issue is probability mass split. In single-answer settings, response consistency (agreement among sampled outputs) is a reliable proxy for confidence; if a model "knows" the answer, outputs converge. However, in multi-answer settings, a model may generate multiple equally correct but textually distinct answers (e.g., "Ibuprofen" and "Acetaminophen" for a headache drug). Existing consistency-based methods interpret this natural diversity as uncertainty or disagreement, leading to a systematic underestimation of confidence. Consequently, models are flagged as unreliable despite being factually correct, causing severe miscalibration. This phenomenon is particularly acute in larger models (e.g., 70B+ parameters) which possess broader knowledge coverage and thus generate a wider variety of correct responses.

Methodology

1. The MACE Benchmark

To systematically study this phenomenon, the authors introduce MACE (Multi-Answer Confidence Estimation), a benchmark comprising 12,000 factual questions across six domains: Honorable Award, Political Office, Regional Affiliation, Mathematical Concept, Natural River, and Linguistic Culture.

• Construction: Questions are generated from Wikidata triplets using controlled templates.
• Cardinality Control: The benchmark explicitly categorizes questions by the number of ground-truth answers: 1, 2, 4, or 6 correct answers (denoted as 1a, 2a, 4a, 6a).
• Quality Control: A rigorous pipeline involving heuristic filtering (popularity and validity) and manual verification by domain experts ensures exhaustive and correct ground-truth sets, addressing limitations in existing benchmarks that lack exhaustive ground truth or controlled cardinality.

2. Experimental Evaluation

The authors evaluated 15 representative calibration methods across four LLM families (LLaMA, Qwen, DeepSeek, and GPT) ranging from 7B to 72B parameters.

• Methods Tested:
  • Single-turn: Token-level probabilities (Entropy, Perplexity), verbalized confidence (Verb), and consistency-based methods (Consistency, Semantic Entropy).
  • Double-turn: Self-evaluation methods (PTrue, Self-Ask) where the model verifies its own answer in a second pass.
• Metrics: Accuracy and AUROC (Area Under the Receiver Operating Characteristic Curve) to measure the quality of confidence estimation.

3. Proposed Solution: Semantic Confidence Aggregation (SCA)

To address the miscalibration caused by probability mass split, the authors propose Semantic Confidence Aggregation (SCA).

• Core Idea: Instead of relying on the single most confident response or penalizing diversity, SCA aggregates the confidence of all high-probability valid answers.
• Mechanism:
  1. Sample multiple responses ($N$) for a question.
  2. Cluster responses based on semantic similarity.
  3. Compute the confidence of each cluster ($C_m$) by summing the token-level generation probabilities of all responses within that cluster: $P(C_m) = \sum_{s \in C_m} p(s)$.
  4. Filter out low-probability clusters (noise) using a threshold $\tau$ (though experiments show $\tau=0$ often suffices).
  5. The final confidence score is the sum of probabilities of the remaining valid clusters: $Conf_{SCA} = \sum_{C_m \in C_{valid}} P(C_m)$.
• Advantage: This approach treats the diversity of correct answers as a feature rather than a bug, preserving high confidence scores even when the model distributes its probability mass across multiple valid semantic clusters.

Key Results

1. The Confidence-Accuracy Paradox

Experiments reveal a counter-intuitive trend: as the number of correct answers increases (from 1a to 6a), QA accuracy improves (questions become empirically easier), but estimated confidence consistently decreases across nearly all existing methods.

• Impact on Larger Models: Larger models (e.g., LLaMA-70B) suffer more severe confidence degradation than smaller models (e.g., LLaMA-8B). This is attributed to "knowledge coverage": larger models internalize more knowledge and thus generate a wider range of valid answers, causing greater probability mass split and lower consistency scores.

2. Failure of Existing Methods

In mixed-answer settings (combining questions with 1, 2, 4, and 6 answers), state-of-the-art (SOTA) calibration methods collapse.

• Consistency-based methods (e.g., Semantic Entropy, Consistency) drop significantly in AUROC because they misinterpret answer diversity as uncertainty.
• Double-turn methods (e.g., PTrue) show greater stability as they decouple verification from generation but still fail to outperform the proposed SCA in overall calibration performance.

3. Performance of SCA

SCA achieves state-of-the-art calibration performance in mixed-answer settings while maintaining strong performance on single-answer questions.

• On the LLaMA-3.1-70B-Instruct model, SCA (with $\tau=0$) achieved an AUROC of 76.7 in the fully mixed setting ([1,2,4,6]), outperforming the next best baseline (Perplexity at 73.8).
• SCA is robust to the threshold $\tau$ and does not require complex clustering or filtering to be effective; simply summing token-level probabilities across all sampled responses often yields optimal results.
• The method generalizes across model scales (7B to 72B) and families (Qwen, LLaMA).

Significance and Claims

The paper claims to advance confidence calibration from the restrictive single-answer QA setting to a more general, realistic scenario involving mixed numbers of ground-truth answers.

• Benchmark Contribution: MACE fills a critical gap in the literature by providing the first benchmark with exhaustive ground truth, multi-answer questions, and controlled answer cardinality, enabling the diagnosis of probability mass split.
• Methodological Insight: The work demonstrates that response inconsistency does not necessarily imply low confidence; in multi-answer contexts, it often reflects the model's ability to access a broader set of valid knowledge.
• Practical Impact: By proposing SCA, the authors provide a training-free, computationally efficient method that prevents the "trust crisis" where correct, diverse responses are incorrectly penalized as unreliable. This is essential for deploying LLMs in real-world applications where questions often have multiple valid solutions.

The authors remain modest regarding limitations, noting that their benchmark uses formulaic prompt templates to control difficulty, which sacrifices naturalistic diversity, and that the current evaluation is restricted to English short-form factual responses. They identify extending to multilingual contexts and long-form generation as necessary future work.