Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong

Technical Summary: Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong

Problem Statement
The evaluation of Large Language Models (LLMs) frequently relies on Multiple Choice Question (MCQ) benchmarks. While accuracy is the standard metric, it fails to capture the model's confidence in its responses. Confidence is often estimated using the probability (logprobs) assigned to the selected token. A critical gap exists in understanding how the answering strategy—specifically, the use of Chain-of-Thought (CoT) prompting versus direct answering—affects these confidence estimates. It is unclear whether reasoning steps genuinely improve the calibration of confidence or if they introduce systematic biases, particularly regarding the reliability of confidence scores for incorrect answers.

Methodology
The authors conducted a comprehensive evaluation across seven LLMs (including Meta's LLaMA3.1-8B and LLaMA3.2-11B, Mistral-7B, Google's Gemma-2-9B, 01.AI's Yi-1.5-9B, and OpenAI's GPT-4o-mini and GPT-4o) using the Massive Multitask Language Understanding (MMLU) benchmark. MMLU covers 57 distinct subjects and over 15,000 questions.

The study compared two prompting conditions for each model:

1. Direct Prompting: The model is instructed to output only the solution letter in a structured format without intermediate text.
2. Chain-of-Thought (CoT) Prompting: The model is instructed to "think step by step" (providing at least three reasoning steps) before outputting the solution letter in the same structured format.

The researchers analyzed the log-probabilities of the selected options to estimate confidence. They employed statistical testing (paired t-tests) to determine significance and utilized standard calibration metrics, specifically Expected Calibration Error (ECE) and Brier scores, to quantify the relationship between confidence and correctness. The analysis was further stratified by correctness (correct vs. incorrect answers) and by subject category.

Key Results
The evaluation yielded five primary findings:

1. Increased Confidence with Reasoning: All tested models exhibited systematically higher confidence (higher assigned probabilities) when using CoT prompting compared to direct answering. This increase was statistically significant ($p < 1 \times 10^{-10}$).
2. Disproportionate Confidence in Errors: While models were generally more confident when correct, the magnitude of the confidence increase was systematically larger when the selected answer was incorrect. CoT prompting amplified high-confidence predictions specifically for wrong answers.
3. Degraded Calibration: The use of CoT prompting worsened standard calibration metrics. For five of the seven models, the Expected Calibration Error (ECE) increased under CoT. Similarly, Brier scores increased for several models, indicating that the higher probabilities assigned to selected options did not translate to better-calibrated predictions.
4. Consistency Across Topics: The confidence inflation occurred across practically all 57 MMLU categories. However, the effect was more pronounced in subjects requiring reasoning (e.g., sciences) compared to fact-based subjects, though even in fact-based subjects where accuracy did not improve (or decreased), confidence in incorrect answers rose significantly.
5. Mechanism of Overconfidence: The study observed that the overconfidence effect is driven by the reasoning process itself, not merely by CoT selecting different options. An ablation study restricting analysis to questions where both prompts selected the same option showed that ECE still increased for all models under CoT.

Significance and Implications
The paper concludes that reasoning processes alter token probabilities, likely because the final answer prediction depends jointly on the question and the model's self-generated reasoning. This leads to inflated confidence estimates that do not necessarily correlate with correctness.

The authors highlight a critical implication for LLM evaluation: in MCQ settings utilizing CoT prompting, logprob-based confidence estimates should be used with extreme caution as a basis for evaluation or metacognitive mechanisms. The phenomenon mirrors human behavior where "explaining is believing," yet in LLMs, this mechanism can lead to a state where the model is highly confident in erroneous conclusions. The study suggests that while CoT improves accuracy in many complex tasks, it simultaneously degrades the reliability of confidence metrics, potentially making erroneous outputs more persuasive and less scrutinized.

The authors note limitations, including the focus on relatively small open-weights models (with the exception of OpenAI's models) and reliance on a single benchmark. They call for future research to explore robust confidence estimation methods that account for response length and CoT depth, and to conduct systematic human-LLM comparative studies to better understand these metacognitive behaviors.