🤖 machine learning

Improved Confidence Estimates for Black-Box Large Language Models

This paper proposes a low-overhead method that improves uncertainty quantification for black-box large language models by training simple classifiers to predict response correctness using existing confidence scores and the correctness of similar queries from a target dataset.

Original authors: Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models have become powerful tools for writing, coding, and answering questions, yet they carry a hidden risk: they sometimes invent facts with absolute certainty. This phenomenon, often called hallucination, creates a significant barrier to using these systems in high-stakes fields like healthcare or scientific research, where a wrong answer can have serious consequences. To solve this, researchers have long tried to build systems that can tell us how much to trust a specific answer. The goal is to assign a score to every response that indicates its reliability. Currently, many methods exist to generate these scores, ranging from asking the model to rate its own confidence to analyzing how much its answers vary when asked the same question in different ways. However, these existing scores often operate in isolation, treating every new question as if it were the first time the model has ever seen it, without looking at how the model has performed on similar questions in the past.

A team of researchers at Layer 6 AI and TD Insurance has proposed a different approach that treats confidence not as a fixed property of the model, but as a pattern that can be learned. They argue that while existing methods generate raw uncertainty scores without needing any prior data, real-world applications almost always involve a dataset of questions and answers that have already been checked for correctness. The researchers realized this labeled data is a vast, untapped resource. Instead of trying to invent a new way to measure uncertainty from scratch, they built a simple system that takes the existing scores and combines them with information about how the model answered similar questions in the past. By training a standard computer program to recognize the relationship between these inputs and the actual correctness of the answers, they created a new method that consistently outperforms the original scores on its own.

The core of their work involves a two-step process that mimics how a human expert might verify a new claim. First, the system takes a new question and the answer the large language model provides. It then calculates the standard uncertainty scores that the model or other tools would normally produce. Next, the system looks into a reference library of previously answered questions to find the ones that are most similar to the new question. It checks whether the model got those similar questions right or wrong in the past and measures how close they are to the current question. These details—the standard scores, the history of similar questions, and the model's past performance on them—are fed into a simple classifier. This classifier learns to predict whether the new answer is correct, effectively translating raw uncertainty signals into a calibrated probability of truth.

The researchers tested this method across several different datasets, including benchmarks for common sense reasoning, scientific facts, and general knowledge questions, using various sizes of language models. They compared their new system against the best existing methods, including those that ask the model to speak its confidence and those that generate many variations of an answer to check for consistency. The results showed that their approach consistently provided more accurate predictions of correctness. In almost every test, the new method was better at distinguishing between right and wrong answers than the original scores alone. Furthermore, the system produced well-calibrated probabilities, meaning that when it predicted an answer had an 80 percent chance of being correct, it was correct roughly 80 percent of the time. This calibration is crucial for decision-making, as it allows users to trust the numbers the system provides.

One of the most significant findings was that the method worked well even when it relied on very little extra information. In some tests, the system used only a single answer from the model and a history of similar questions, without needing the model to generate multiple variations of the answer. This kept the computational cost low, making the method practical for real-world use where speed and efficiency matter. The researchers also found that the best way to combine these signals depended on the specific task, suggesting that a flexible approach that can adapt to different types of questions is superior to a one-size-fits-all solution. By treating confidence estimation as a learning problem rather than a fixed calculation, the team demonstrated that existing data can be used to significantly improve the safety and reliability of large language models without requiring access to their internal workings or expensive retraining.

The study concludes that while the initial uncertainty scores generated by current tools are useful, they are not the final word on reliability. By leveraging the simple fact that models tend to perform consistently on similar types of questions, a supervised learning framework can refine these scores into much more trustworthy estimates. This does not replace the need for good uncertainty metrics but rather enhances them, turning raw data into actionable insight. The work suggests that the path to safer artificial intelligence lies not just in building better models, but in building better ways to understand and interpret the answers those models give.

Technical Summary: Improved Confidence Estimates for Black-Box Large Language Models

Problem Statement

The safe deployment of Large Language Models (LLMs) in high-stakes environments (e.g., healthcare, scientific research) is hindered by their susceptibility to hallucinations. While Uncertainty Quantification (UQ) aims to evaluate the reliability of model outputs, existing methods face significant limitations:

Zero-Shot Limitation: Most current UQ methods (heuristic scoring or verbalized confidence) operate in a zero-shot manner, generating uncertainty scores without leveraging available labelled evaluation data.
Lack of Calibration: Existing scores often correlate with correctness but fail to provide calibrated probabilities. For instance, verbalized confidence ( $P_{true}$ ) is frequently overconfident, and heuristic scores (e.g., semantic entropy) are abstract values rather than direct probability estimates.
Decision-Making Gap: Heuristic scores are primarily designed for abstention (withholding a response if a threshold is met). However, the authors argue that confidence estimation—providing a calibrated probability of correctness—is superior for downstream decision-making, as it allows systems to maximize expected utility within decision-theoretical frameworks.

The core problem addressed is how to transform existing, often uncalibrated, zero-shot uncertainty scores into robust, calibrated confidence estimates by leveraging the labelled datasets that are standard prerequisites for model evaluation.

Methodology

The authors propose a supervised learning framework that treats existing UQ scores as features rather than final outputs. The method does not require white-box access to the LLM or additional model generations beyond what the baseline scores require.

1. Data Partitioning

The available labelled dataset $\mathcal{D}$ is partitioned into:

Training Set ( $\mathcal{D}_{train}$ ): Used to fit the confidence classifier.
Reference Set ( $\mathcal{D}_{ref}$ ): Used to construct auxiliary features by summarizing the LLM's historical performance on similar queries.

2. Feature Engineering

For a given query-response pair $(q, r)$ , the feature vector $X(q, r)$ is constructed from two components:

Baseline Uncertainty Scores ( $S$ ): A collection of $c$ pre-existing scores (e.g., verbalized $P_{true}$ , semantic entropy, Laplacian approximation). These are treated as raw inputs.
Neighborhood-Based Features ( $f_{ref}$ ): To capture local empirical patterns, the method retrieves the $k$ $k$ -nearest neighbors of $q$ $q$ from the reference set using cosine similarity in an embedding space. From these neighbors, the following statistics are extracted:
- The vector of correctness labels ( $y_i$ ) for the neighbors.
- The empirical mean ( $\mu_y$ ) and standard deviation ( $\sigma_y$ ) of neighbor correctness.
- The vector of cosine similarities between $q$ and its neighbors.
- The empirical mean ( $\mu_{sim}$ ) and standard deviation ( $\sigma_{sim}$ ) of these similarities.

The final feature vector is $X(q, r) = (S(q, r), f_{ref}(q))$ .

3. Supervised Classification

The problem is framed as a binary classification task to predict the probability of correctness $Y=1$ .

Model Selection: The authors employ an Auto-ML approach to select the optimal classifier for a specific task. Candidates include $\ell_1$ -regularized logistic regression and Random Forests.
Training: Models are trained to minimize cross-entropy loss on $\mathcal{D}_{train}$ .
Calibration: A post-hoc temperature scaling step is applied to ensure the output probabilities are well-calibrated (minimizing Expected Calibration Error, ECE).

4. Inference

At inference time, the process involves:

Computing baseline scores $S(q, r)$ .
Retrieving $k$ -nearest neighbors from $\mathcal{D}_{ref}$ .
Evaluating the trained classifier $f_\theta$ on the combined feature vector.
The computational overhead is negligible compared to the cost of generating LLM responses.

Key Contributions

Paradigm Shift: The paper argues that UQ for LLMs should be treated as a supervised learning problem that leverages available labelled data, moving beyond the zero-shot paradigm.
General-Purpose Framework: The method acts as a "wrapper" that can aggregate and refine any existing uncertainty score (heuristic or verbalized) into a calibrated confidence estimate without modifying the underlying LLM.
Neighborhood Statistics: The introduction of neighborhood-based features (correctness and similarity statistics from a reference set) provides a novel way to incorporate empirical evidence of model reliability into the prediction.
Task-Adaptive Learner: The authors demonstrate that the optimal classifier model is task-dependent, necessitating an Auto-ML approach to select between linear and non-linear models based on the data distribution.

Experimental Results

The framework was evaluated across four datasets (CommonSense QA, Natural Questions, SciQ, SimpleQA) and three GPT-4.1 model variants (flagship, mini, nano), as well as the open-source LLaMA4Maverick.

Performance: The proposed Auto-ML method consistently outperformed all baselines in Area Under the Receiver Operating Characteristic Curve (AUROC). It achieved the highest AUROC in nearly all scenarios, with only one minor exception where the difference was negligible.
Calibration: The method produced significantly lower Expected Calibration Error (ECE) compared to raw verbalized confidence ( $P_{true}$ ) and often outperformed temperature-scaled versions ($ts$- $P_{true}$ ).
Ablation Studies:
- Without Semantic Features: Even when excluding complex semantic uncertainty scores (relying only on $P_{true}$ and neighborhood statistics), the method maintained competitive performance with lower computational cost.
- Model Family: The approach generalized well to open-source models (LLaMA4Maverick), confirming its applicability beyond black-box APIs.
Baseline Comparison: The method significantly outperformed APRICOT (a similar classifier-based approach) and various semantic-based heuristic scores (semantic entropy, Laplacian, kernel entropy), particularly on multiple-choice datasets where semantic methods struggled.

Significance and Claims

The paper claims that labelled evaluation data is a significant, yet untapped resource for improving LLM uncertainty estimates. The authors assert that:

Reliability: By moving beyond zero-shot heuristics, one can consistently produce more accurate and calibrated confidence estimates.
Efficiency: The method introduces minimal computational overhead, making it a "cheap and straightforward enhancement" for real-world applications.
Flexibility: It serves as a general mechanism to transform heuristic scores into decision-ready probabilities, bridging the gap between abstract uncertainty metrics and practical decision-making.
Future Direction: While the current work focuses on binary correctness, the authors suggest this paradigm could be extended to multi-class classification (e.g., distinguishing between "partially correct" and "excellent" responses) to further refine LLM deployment safety.

The authors maintain a modest stance, acknowledging that their approach requires training dataset-specific classifiers and relies on the quality and size of the reference set for effective retrieval, though they argue these are standard requirements in practical model development lifecycles.