Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models

Technical Summary: Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models

Problem Statement
Large Language Models (LLMs) frequently generate confident but incorrect answers rather than abstaining when uncertain. This issue is particularly acute for Small Language Models (SLMs), defined here as transformer-based generative models with fewer than 10 billion parameters. SLMs are increasingly deployed in edge and mobile environments where computational constraints and autonomous operation (often without immediate human oversight) necessitate reliable uncertainty detection. Existing methods for enabling abstention face significant limitations: training-based approaches require access to model weights and retraining; prompt-based methods relying on self-reported confidence often fail as models do not accurately self-assess; and sampling-based consensus methods incur prohibitive computational overhead unsuitable for resource-constrained SLMs.

Methodology: Second Guess
The paper proposes "Second Guess," a lightweight, parameter-free prompting technique designed for multiple-choice question answering (MCQA) that requires no access to internal model states or parameters. The method operates on the empirical insight that models with stable knowledge will provide consistent answers regardless of the available options, whereas uncertain models exhibit unstable behavior when an "I don't know" option is introduced.

The technique involves two forward passes per question:

1. Original Prompt: The model is prompted with the standard set of four answer choices ($C = \{c_1, c_2, c_3, c_4\}$), yielding response $R_1$.
2. Augmented Prompt: An "I don't know" option is inserted at a random position into the choice set ($C' = \{c_1, c_2, c_3, c_4, \text{"I don't know"}\}$), and the model is prompted again to yield response $R_2$.

The final output $R$ is determined by comparing the two responses:

• If $R_1 = R_2$, the original answer is retained.
• If $R_1 \neq R_2$ (regardless of what the new answer is), the output is classified as an abstention ("I don't know").

This approach effectively treats answer discrepancies between the two prompts as a signal of uncertainty, requiring only two queries per question.

Experimental Setup
The authors evaluated Second Guess on four instruction-tuned SLMs (ranging from 2B to 8B parameters: Mistral-7B, Llama-3.1-8B, IBM Granite-3.3-2B, and Qwen3-4B) across four MCQA benchmarks: CommonsenseQA, QASC, MMLU-Pro, and SuperGPQA. The evaluation included both base models and models fine-tuned using Low-Rank Adaptation (LoRA).

The method was compared against several baselines:

• Original Prompt: Standard MCQA without augmentation.
• Augmented Prompt: A single-pass approach where "I don't know" is included from the start.
• Self-Evaluation: A two-step strategy where the model verifies its own answer.
• Entropy Thresholding: A post-hoc method converting high-entropy outputs to abstentions.

Key Results
The evaluation utilized three metrics: Precision, Error Rate, and Composite Risk (a combination of incorrect answers and cases where the model abstains despite having the correct answer originally).

1. Composite Risk Reduction: Second Guess achieved the highest composite risk improvement of 10.81% across all base models and datasets, outperforming the best baseline (Self-Evaluation at 7.69%) while using the same number of queries (two).
2. Robustness to Fine-Tuning: On fine-tuned models, Second Guess maintained an 8% composite risk improvement. In contrast, entropy-based methods degraded significantly on fine-tuned models because increased model confidence rendered token entropy less informative for uncertainty detection.
3. Inverse Relationship with Base Performance: The paper identifies an inverse relationship between a model's base accuracy and the improvement gained from Second Guess. Models with lower base accuracy (e.g., Llama on SuperGPQA with ~32% accuracy) saw the most significant risk reductions (exceeding 25%), while high-performing models saw modest or no gains.
4. Mechanism of Improvement: Analysis of answer transitions revealed that the risk reduction primarily stems from detecting option switching rather than the explicit selection of "I don't know." Models that were uncertain tended to switch between incorrect options when the "I don't know" option was added, rather than consistently selecting the abstention token.

Significance and Claims
The paper claims that Second Guess offers a practical, accessible solution for improving the reliability of SLMs in safety-critical and autonomous contexts. Its significance lies in being:

• Parameter-free: It requires no hyperparameter tuning or access to model weights.
• Computationally Efficient: It demands only two forward passes, making it viable for black-box SLMs and edge devices where sampling-based methods are too costly.
• Effective for Lower-Performing Models: It specifically targets the reliability gap in smaller, less capable models where error reduction is most critical.

The authors note limitations, acknowledging that the method's utility diminishes as model scale and base performance increase, and that the evaluation did not include Chain-of-Thought (CoT) reasoning, which may alter baseline capabilities. However, the work establishes that behavioral instability (answer switching) is a robust, detectable signal of uncertainty that can be leveraged to induce abstention without complex architectural changes.