Entropy Alone is Insufficient for Safe Selective Prediction in LLMs

1. Problem Statement

Large Language Models (LLMs) frequently produce "hallucinations"—fluent but factually incorrect answers—often with high confidence. In high-stakes domains (e.g., healthcare), Selective Prediction systems are proposed to mitigate this by abstaining from answering when uncertainty is high.

However, current uncertainty quantification (UQ) methods face two critical issues:

1. Model-Dependent Failure: Entropy-based methods (which measure the spread of the model's output distribution) often fail to detect a specific failure mode known as the "confidently wrong" regime. In this regime, models generate low-entropy (highly confident) hallucinations, causing entropy-based risk scores to incorrectly classify them as safe.
2. Misaligned Evaluation: Prior work evaluates UQ methods using global detection metrics like AUROC. These metrics can obscure failures that are critical in deployment, specifically the inability to maintain low error rates at strict safety thresholds.

The paper argues that relying solely on entropy is insufficient for safe selective prediction and that evaluation must shift toward deployment-facing metrics that reflect real-world risk control.

2. Methodology

2.1 Uncertainty Signals

The authors evaluate and combine two distinct types of uncertainty signals:

• Entropy-Based Signals ($u_{ent}$):
  • Sequence NLL: Negative log-likelihood of the generated answer (single-pass).
  • Semantic Entropy (SE): Clusters multiple sampled completions based on semantic entailment to measure dispersion.
  • Semantic Entropy Probe (SEP): A lightweight linear probe trained to predict binarized SE, amortizing the cost of sampling.
• Correctness-Based Signal ($u_{pc}$):
  • Probability of Correctness (PC) Probe: A logistic regression classifier trained on prompt representations to predict whether an answer is correct (using external ground truth labels). This signal is converted to a risk score ($1 - p_{pc}$).

2.2 The Combined Approach

The core proposal is a hybrid risk score generated by a two-feature logistic regression that fuses the entropy signal ($u_{ent}$) and the PC probe signal ($u_{pc}$). This combined score is used to determine whether to answer or abstain based on a threshold $\tau$.

2.3 Evaluation Framework

The study moves beyond standard classification metrics to Selective Prediction metrics:

• Risk-Coverage (RC) Curves: Plotting the hallucination rate (risk) against the fraction of questions answered (coverage).
• Excess Area Under the Risk-Coverage Curve (E-AURC): Measures how close the system is to an "oracle" that abstains perfectly.
• Target Calibration Error (TCE): Measures the deviation between the target risk level (e.g., "I want to answer only when risk < 5%") and the realized risk on the test set. This is crucial for ensuring the system behaves as promised at specific safety thresholds.

2.4 Experimental Setup

• Datasets: TriviaQA (general knowledge), BioASQ (biomedical), and MedicalQA (open-ended medical).
• Models: Four model families (Mistral, Llama, Qwen, Gemma) ranging from 3B to 8B parameters.
• Protocol: Models generate answers; probes are trained on a split of the data; thresholds are tuned on a calibration set to meet specific target risks ($\alpha \in [0.05, 0.30]$).

3. Key Contributions

1. Identification of the "Confidently Wrong" Regime: The authors demonstrate that entropy-based methods suffer from a model-dependent failure mode where models produce low-entropy hallucinations. For example, on the Qwen model, many hallucinations collapse to near-zero semantic entropy, making them indistinguishable from correct answers using entropy alone.
2. Complementarity of Signals: They show that entropy and correctness probes are only moderately correlated (Spearman $\rho \approx 0.5$). The PC probe successfully assigns high risk scores to the "confidently wrong" instances that entropy misses, while entropy captures uncertainty in other failure modes.
3. Superiority of Combined Methods: The fusion of entropy and correctness signals consistently outperforms entropy-only baselines across all datasets and models.
4. Shift in Evaluation Paradigm: The paper advocates for deployment-facing evaluation. It demonstrates that a method with high AUROC (good global detection) can still fail catastrophically at strict safety thresholds (high TCE), whereas the combined method maintains reliable calibration.

4. Key Results

• Performance Gains: Across three benchmarks and four model families, the combined method (Entropy + PC Probe) generally improves both the risk-coverage trade-off (lower E-AURC) and calibration (lower TCE) compared to entropy-only baselines.
• Failure of Entropy-Only in High-Trust Regimes:
  • In the "High Trust Regime" (target risk $\leq 0.15$), entropy-based methods often hit a "risk floor," meaning they cannot reduce the hallucination rate below a certain level without abstaining on almost all questions (zero coverage).
  • The combined method lowers this risk floor, allowing the system to operate safely at stricter targets with non-trivial coverage.
• Model Dependence: The effectiveness of individual methods varies by model. For instance, Semantic Entropy (SE) works best for Llama and Mistral on TriviaQA, while the PC Probe dominates for Qwen. This reinforces the need for combined approaches to ensure robustness across different architectures.
• Calibration: The combined method tracks the ideal calibration diagonal much more closely than entropy-only methods when tuned to strict targets, whereas entropy-only methods diverge sharply, leading to uncalibrated risk estimates.

5. Significance and Implications

• Safety in Deployment: The findings suggest that for safety-critical applications (like medical diagnosis), relying solely on entropy is dangerous. Systems must incorporate correctness probes or similar signals to detect "confidently wrong" hallucinations.
• Evaluation Standards: The paper challenges the community to stop relying solely on AUROC for UQ research. Instead, metrics like TCE and E-AURC should be standard, as they directly answer the question: "Can we trust this system to operate at a stated risk level?"
• Efficiency: The proposed solution (a lightweight linear probe combined with entropy) does not require additional inference overhead beyond a single forward pass (for the probe) and offers a practical path to safer LLM deployment without the heavy computational cost of full Semantic Entropy sampling at test time.

Limitations

The authors note that the study is limited to short-form QA tasks. It remains unclear if the "confidently wrong" failure mode or the complementarity of signals generalizes to long-form generation or complex reasoning tasks. Additionally, the PC probe requires supervised training on labeled data, which may be a bottleneck in low-resource settings.