Technical Summary: Zero-Shot Confidence Estimation for Small LLMs

Problem Statement

The reliable deployment of small language models (SLMs, 7–8B parameters) for local-to-cloud routing hinges on accurate confidence estimation. The goal is to route easy queries to a cheap local model and escalate difficult ones to an expensive cloud model. The dominant approach involves training a supervised classifier (e.g., RouteLLM) on labeled data to predict model success based on query embeddings. However, this approach suffers from brittleness: supervised estimators learn properties of the specific query distribution (embedding regions corresponding to "easy" questions) rather than the model's internal capability. Consequently, these estimators fail to transfer when query distributions shift, new domains are introduced, or in cold-start scenarios where no labeled data exists.

Methodology

The paper evaluates zero-shot confidence signals against supervised baselines across three 7–8B model families (Qwen-2.5-7B, Llama-3.1-8B-Instruct, Mistral-7B-Instruct) and two datasets (MMLU-Pro and TriviaQA).

Signals Evaluated

1. Zero-Shot Signals (No Training Data):

   • Log-probability (logprob): The average per-token log-probability of the generated answer, mapped to [0, 1]. This measures the model's token-level surprise.
   • Self-Assessment (GSA): A single-token probe asking "Are you confident?" where the score is the normalized probability of the "YES" token.
   • Retrieval-Conditional Self-Assessment (GSA v3): A novel pre-generation signal that injects retrieved knowledge only when similarity scores exceed a threshold ($\tau=0.70$). If no strong hits are found, the prompt remains byte-identical to the bare prompt to prevent the model from being primed toward "NO" due to the absence of context.
   • Self-Consistency (SC): Jaccard similarity between two independent generations.
   • Knowledge Similarity (KS): Cosine similarity between the query and retrieved knowledge base entries.

2. Supervised Baselines:

   • RouteLLM-nm: Logistic regression trained on query embeddings to predict local sufficiency.
   • RouteLLM-pks: Logistic regression trained on query embeddings concatenated with knowledge similarity.
   • Training was performed on 1,000 labeled examples per model.

Experimental Setup

• Datasets: MMLU-Pro (10-option MCQ, ~950 queries/model) and TriviaQA (open-ended, 500 queries/model).
• Evaluation: Area Under the Receiver Operating Characteristic curve (AUROC) with 1,000-sample bootstrap confidence intervals.
• Judge: Claude Opus 4.5 used for labeling, with fallback rates tracked.

Key Results

1. In-Distribution Performance

On MMLU-Pro, log-probability emerged as the best single zero-shot signal, achieving AUROCs of 0.714 (Qwen), 0.650 (Llama), and 0.678 (Mistral). It matched or exceeded supervised baselines (RouteLLM-nm/pks), which ranged from 0.644 to 0.676. Notably, a supervised baseline trained on 1,000 examples never consistently exceeded the zero-shot log-probability signal.

2. Out-of-Distribution (Cross-Dataset) Transfer

The most significant finding is the divergence in transferability:

• Supervised Baselines Collapse: When models trained on MMLU-Pro were tested on TriviaQA, RouteLLM performance dropped to chance levels (AUROC 0.512–0.564).
• Zero-Shot Signals Improve: Log-probability performance increased on TriviaQA, reaching AUROCs of 0.828 (Qwen), 0.800 (Llama), and 0.717 (Mistral).
• Explanation: Supervised signals learn query-side properties (embedding regions) specific to the training distribution. Zero-shot signals measure generation-side properties (token-level surprise), which are intrinsic to the model and independent of the input distribution.

3. Learning Curves and Latency

• Learning Curve: Supervised routing is unstable at small sample sizes (N=25) and converges around N=250–500, but even at N=1,000, it does not surpass the zero-shot baseline.
• Latency: Log-probability requires full generation (1.5–4.8s). GSA v3 acts as a pre-generation filter with significantly lower latency (0.5–0.8s), offering a 3–10× speedup over log-probability while improving over bare self-assessment by up to +0.069 AUROC.

4. Signal Fusion

Naive fusion (simple averaging) of multiple signals generally degraded performance, dragging strong signals (log-probability) down toward weaker ones (e.g., Knowledge Similarity, which was consistently inverted).

Significance and Claims

The paper claims that average token log-probability is a robust, zero-cost alternative to supervised routing for factual QA tasks. Its primary significance lies in demonstrating that:

1. Generation-side signals transfer; query-side signals do not: The mechanistic distinction between measuring the model's generation process versus the query distribution explains why supervised routing fails under distribution shift.
2. Supervision is often unnecessary: For 7–8B models on factual tasks, training a classifier on 1,000 labeled examples provides no advantage over a simple zero-shot log-probability check and fails completely when the domain changes.
3. Retrieval-Conditional Self-Assessment: This novel pre-generation signal offers a low-latency alternative that selectively grounds confidence judgments without introducing negative priming effects.

Limitations

The authors explicitly note the scope is limited to:

• Two factual QA datasets (MMLU-Pro, TriviaQA); reasoning, creative, and multi-turn tasks are untested.
• Three models of similar scale (7–8B); the hypothesis that log-probability improves with scale is untested.
• Static knowledge bases; production environments with growing KBs were not simulated.
• Label noise (~13% inter-judge disagreement), though differential comparisons remain robust.

The paper concludes that for practitioners deploying local-to-cloud routing on factual QA, log-probability should be the default starting point, with GSA v3 as a low-latency pre-filter, reserving supervised training only for scenarios with stable, known distributions where pre-generation routing is strictly required.