Query-Level Uncertainty in Large Language Models

This paper proposes "Internal Confidence," a novel, training-free method that estimates query-level uncertainty by leveraging self-evaluations across model layers and tokens to detect knowledge boundaries, thereby enabling more efficient and trustworthy adaptive inference strategies like retrieval-augmented generation and model cascading.

The Gist

Imagine you have a brilliant, encyclopedic friend who knows almost everything. But sometimes, they get a little too confident and start making things up (hallucinating), or they waste hours trying to solve a simple math problem they could have answered instantly.

This paper introduces a new "gut feeling" system for Large Language Models (LLMs) called Internal Confidence. It's a way for the AI to know, before it even starts typing an answer, whether it actually knows the answer or if it's just guessing.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Blind Guess" vs. The "Smart Pause"

Currently, most AI uncertainty checks happen after the AI has written a long answer. It's like a student writing a whole essay, then the teacher grading it and saying, "Actually, you didn't know this." By then, the student has wasted time and energy.

• Old Way (Answer-Level Uncertainty): The AI writes a 500-word answer, then checks if it's confident. If it's not, it deletes the essay and tries again. Waste of time and money.
• New Way (Query-Level Uncertainty): The AI looks at the question, pauses for a split second, and says, "I know this!" or "I have no idea." Zero wasted time.

2. The Solution: The "Internal Gut Check"

The authors created a method called Internal Confidence. Instead of waiting for the AI to write an answer, they peek inside the AI's "brain" (its internal layers) while it is just reading the question.

The Analogy: The Library of Babel
Imagine the AI is a massive library.

• The Old Method: You ask the librarian for a book. They pull out a random book, read it, write a summary, and then realize, "Oh, this book is about the wrong topic!"
• The New Method: You ask the librarian the question. Before they even walk to the shelves, they check their internal map. They feel a "vibe" (a mathematical signal) that says, "I know exactly where this book is," or "This book doesn't exist in our library."

How it works technically (simplified):
The researchers ask the AI a simple "Yes/No" question: "Can you answer this?"
They don't wait for the AI to say "Yes." Instead, they look at the tiny electrical signals in the AI's brain as it processes that question. They measure the probability of the AI thinking "Yes" across all its different layers of thinking. They combine these signals into one score: Internal Confidence.

3. Why is this a Game Changer?

The paper shows three major benefits:

A. Speed: The "Lightning Bolt" vs. The "Slow Walk"

Existing methods are slow because they require the AI to generate text first.

• Analogy: Imagine trying to check if a car is fast by driving it 100 miles. That takes time.
• The New Method: This is like checking the engine's RPM while the car is still in the garage. It's 30 to 600 times faster. The AI can decide in a fraction of a second if it needs help.

B. Saving Money: The "Smart Switch"

LLMs cost money to run. If you have a simple question, you don't need the most expensive, powerful AI.

• The Scenario: You have a small, cheap AI and a big, expensive AI.
• The Old Way: You send every question to the big AI just to be safe.
• The New Way: The small AI uses its "Internal Confidence" to check the question.
  • High Confidence? "I got this!" (Saves money).
  • Low Confidence? "Pass this to the big boss." (Saves accuracy).
    This is called Model Cascading. It's like a receptionist who handles simple calls but immediately transfers complex ones to the manager, saving the manager's time.

C. Trust: Knowing When to Say "I Don't Know"

In high-stakes fields like medicine or law, it's dangerous for an AI to guess.

• The Analogy: A doctor who says, "I'm not sure, let me check the textbook," is better than a doctor who confidently prescribes the wrong medicine.
• The Benefit: This method allows the AI to confidently say, "I don't know," before it hallucinates a fake fact. It can then trigger a RAG (Retrieval-Augmented Generation) system to go look up the answer in a database, ensuring the final answer is true.

4. The "Sweet Spot"

The researchers found a "Goldilocks Zone." By adjusting the threshold of how confident the AI needs to be before answering, you can find a perfect balance where you save the most money and time without losing any accuracy.

Summary

This paper teaches AI to know what it knows before it starts talking.

• Before: AI guesses, writes a long answer, then realizes it was wrong. (Slow, expensive, risky).
• After: AI checks its "gut feeling," decides if it knows the answer, and only then proceeds. (Fast, cheap, safe).

It's like giving the AI a pair of glasses that lets it see the boundaries of its own knowledge, so it never wastes time trying to solve a puzzle it doesn't have the pieces for.

Technical Summary

Here is a detailed technical summary of the paper "Query-Level Uncertainty in Large Language Models" (ICLR 2026).

1. Problem Statement

Large Language Models (LLMs) possess inherent knowledge boundaries; they cannot accurately answer every query. Current uncertainty estimation methods primarily focus on Answer-Level Uncertainty, which evaluates the reliability of a response after it has been generated. This approach incurs significant computational costs, especially for long or complex answers, and fails to prevent the generation of hallucinations or irrelevant content in the first place.

The paper addresses the gap in Query-Level Uncertainty: the ability to determine before generating any tokens whether an LLM possesses the necessary knowledge to answer a specific query. The core research question is: Can we detect if a model can address a query based solely on its internal states during the forward pass of the input prompt, without generating an answer?

2. Methodology: Internal Confidence

The authors propose Internal Confidence (IC), a novel, training-free, and generation-free method to estimate query-level uncertainty.

Core Concept

Instead of generating an answer and then evaluating it, IC leverages the LLM's internal self-evaluation capabilities. The method is grounded in the observation that LLMs can implicitly assess their own knowledge boundaries through a single forward pass.

Technical Implementation

1. Binary Self-Assessment Prompt: The model is prompted with a yes/no question: "Respond only with 'Yes' or 'No' to indicate whether you are capable of answering the {Query} accurately."
2. Layer and Token Aggregation:
   • Unlike previous methods that only look at the final token in the final layer, IC computes the probability of the token YES ($P(\text{YES})$) across all layers ($l$) and all token positions ($n$) of the input query.
   • This results in a matrix of probabilities $P(\text{YES} | h^{(l)}_n)$, where $h^{(l)}_n$ is the hidden state.
3. Decision Center & Attenuated Encoding:
   • Empirical analysis shows that the ability to distinguish answerable from non-answerable queries is not uniform; it peaks at a specific "decision center" (typically near the last layer and last token, though it varies).
   • To aggregate these signals effectively, IC uses Attenuated Encoding. It applies a weighted sum where weights decay based on the distance from the decision center.
   • The formula for Internal Confidence is:
     $$IC(h) = \sum_{n=1}^{N} \sum_{l=1}^{L} w^{(l)}_n P(\text{YES} | h^{(l)}_n)$$
   • The weights $w^{(l)}_n$ are derived from a Gaussian-like decay function controlled by a locality parameter $\alpha$, allowing the model to focus on the most informative internal states while smoothing out noise.

3. Key Contributions

• Conceptual Shift: Introduces the formal definition of Query-Level Uncertainty, shifting the paradigm from post-generation reliability to pre-generation capability assessment.
• Training-Free Efficiency: Proposes Internal Confidence, a method that requires no fine-tuning, no additional training data, and no answer generation. It operates with a single forward pass.
• Novel Aggregation Strategy: Demonstrates that aggregating self-evaluation signals across multiple layers and tokens using attenuated encoding significantly outperforms naive averaging or single-point extraction.
• Adaptive Inference Framework: Shows how IC can serve as a gating mechanism for adaptive inference strategies like Retrieval-Augmented Generation (RAG) and Model Cascading.

4. Experimental Results

The authors evaluated Internal Confidence on three datasets: TriviaQA and SciQ (factual QA) and GSM8K (mathematical reasoning), using models of varying sizes (Phi-3.8B, Llama-8B, Qwen-14B).

• Accuracy (AUROC & PRR):
  • Internal Confidence consistently outperformed state-of-the-art baselines (e.g., Perplexity, Semantic Entropy, P(TRUE), MSP) in distinguishing answerable vs. non-answerable queries.
  • On Qwen-14B, IC achieved an average AUROC of 67.1 and PRR of 31.7, surpassing all other methods.
  • It also demonstrated superior calibration (lower Expected Calibration Error) compared to baselines.
• Computational Efficiency:
  • Speed: IC is drastically faster than answer-level methods. While methods like Semantic Entropy or SAR require generating full answers (taking 150s–180s per sample), IC takes only 0.3 seconds.
  • Speedup: This represents a 30x to 600x speedup compared to existing baselines.
  • Scalability: The runtime of IC is constant regardless of answer length, whereas answer-level methods scale linearly with answer length.
• Adaptive Inference Applications:
  • Efficient RAG: Using IC as a threshold to decide when to invoke RAG reduced inference costs significantly while maintaining accuracy. The authors identified an "optimal point" where resource usage is minimized without performance loss.
  • Model Cascading: IC successfully guided a small model (Phi-3.8B) to defer difficult queries to a larger model (Llama-8B), optimizing the cost-performance trade-off.

5. Significance and Impact

• Cost Reduction: By preventing the generation of answers for queries the model cannot answer, IC saves substantial computational resources and monetary costs, which is critical for agentic workflows and large-scale deployments.
• Trustworthiness: It enables models to adopt an "abstention strategy" (saying "I don't know") before hallucinating, which is vital for high-stakes domains like healthcare and law.
• Generalizability: The method works across different model architectures and sizes without requiring model-specific fine-tuning, making it a robust baseline for uncertainty estimation.
• Future Direction: The paper establishes a strong foundation for "pre-generation" uncertainty, suggesting that future research should focus on refining decision centers and extending these concepts to black-box APIs (where internal states are inaccessible).

In conclusion, Internal Confidence provides a highly efficient, accurate, and training-free mechanism for LLMs to self-assess their knowledge boundaries, enabling smarter, faster, and more reliable adaptive inference systems.