💬 NLP

How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality

This paper systematically investigates how LLMs utilize reasoning chains to judge answer factuality, revealing that while reasoning can aid strong judges, both weak and strong models remain susceptible to being misled by fluent but incorrect reasoning, highlighting the critical need for more robust evaluation metrics that distinguish genuine reasoning quality from superficial fluency.

Original authors: Minzhu Tu, Shiyu Ni, Keping Bi

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minzhu Tu, Shiyu Ni, Keping Bi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Teacher" and the "Student"

Imagine you are a teacher grading a student's math test.

The Student is an AI model that answers questions.
The Teacher is another AI model (the "Judge") that decides if the student got the answer right or wrong.

In the past, teachers only looked at the final answer (e.g., "The answer is 42"). But now, students are allowed to show their work (their "reasoning chain") before writing the final number.

This paper asks a simple but tricky question: Does seeing the student's "work" help the teacher grade better, or does it just trick them?

The Main Findings: The "Fluency Trap"

The researchers found that showing the "work" changes how the teacher grades, but not always in a good way. It depends on how smart the teacher is.

1. The "Weak Teachers" (Smaller AI Models)

Think of a weak teacher as someone who is easily impressed by a neat handwriting or a long, confident essay.

What happens: When the student writes a long, smooth-sounding explanation (even if the logic is wrong), the weak teacher gets fooled. They think, "Wow, this explanation sounds so professional and fluent! The answer must be right!"
The Result: They give a passing grade to wrong answers just because the "reasoning" sounded good. It's like a student writing a beautiful poem about why $2+2=5$ , and the teacher giving them an A because the poem was so well-written.

2. The "Strong Teachers" (Larger, Smarter AI Models)

Think of a strong teacher as a strict professor who actually reads the math.

What happens: These teachers are better at spotting errors. If the student's work has a mistake in the middle, the strong teacher catches it and marks the answer wrong.
The Catch: Even the best teachers can be fooled. If the student (especially a very advanced one like DeepSeek-V3.1) writes a reasoning chain that looks perfectly logical and high-quality, even the strong teacher might get tricked into thinking a wrong answer is right.

The Experiment: Breaking the "Reasoning"

To figure out why the teachers were getting fooled, the researchers played a game of "spot the difference." They took the student's reasoning and secretly changed two things:

A. The "Fluency" Test (The Flow)

They took a perfect explanation and inserted random, boring facts that had nothing to do with the question (e.g., "The Earth orbits the Sun in 365 days" inserted into a math problem).

Result: The moment the flow was broken, all the teachers (weak and strong) suddenly became suspicious. They stopped giving passing grades.
Lesson: The teachers love it when the explanation flows smoothly. If the flow is choppy, they assume the answer is wrong.

B. The "Factuality" Test (The Truth)

They took those random facts and changed them to be lies (e.g., "The Earth orbits the Sun in 100 days").

Result: This made the teachers even more likely to reject the answer.
Lesson: Teachers care about whether the facts inside the reasoning are true. If the reasoning contains lies, the answer is likely wrong.

C. The "Position" Test (Where the lie is)

They put the fake facts at the beginning of the explanation vs. the end.

Result: If the lie was at the start, the teachers immediately rejected the whole thing. If the lie was at the end, the teachers were more likely to overlook it.
Analogy: It's like a story. If the first sentence of a story is nonsense, you stop reading. If the nonsense is just a weird footnote at the end, you might still think the story was good.

The Takeaway: A Double-Edged Sword

The paper concludes that "Reasoning" is a double-edged sword for AI evaluation:

It helps: Sometimes, seeing the work helps a smart teacher catch a mistake they would have missed otherwise.
It hurts: Often, it tricks teachers into thinking a wrong answer is right just because the explanation sounded confident and fluent.

The Bottom Line:
We need to build "super-teacher" AIs that don't just get impressed by smooth talking or long explanations. They need to be able to look past the fancy words and check if the logic is actually true, even if the student is trying to sound smart. Until then, seeing the "work" might actually make our grading system less reliable, not more.

1. Problem Statement

Large Language Models (LLMs) are increasingly used as "LLM-as-a-Judge" to evaluate the correctness of other models' outputs, serving as a scalable alternative to human evaluation. However, these judges are known to be susceptible to surface-level biases (e.g., fluency, length).

With the rise of reasoning-capable models (e.g., Chain-of-Thought, CoT), judges now have access not just to the final answer but also to the intermediate reasoning traces. The core research questions are:

Does exposing the reasoning process to the judge improve judgment accuracy?
How do models of varying capabilities (weak vs. strong) utilize this additional information?
What specific attributes of reasoning chains (fluency vs. factuality) drive the judge's decision?

2. Methodology

Experimental Setup

Generators: The study uses a mix of open-source models (Qwen3 series: 8B, 14B, 32B; Llama-3; GLM-4) and closed-source models (DeepSeek-V3.1, GPT-4o, Claude Sonnet 4.5) to generate answers.
Judges: A diverse set of models ranging from "weak" (smaller open-source) to "strong" (large open-source and top-tier closed-source) act as evaluators.
Datasets: Experiments cover two factual QA datasets (Natural Questions, HotpotQA) and two mathematical reasoning datasets (GSM8K, MATH-500).
Settings:
- Judge-Answer-Only: The judge sees only the final answer ( $a$ ).
- Judge-Answer-and-Reasoning: The judge sees the reasoning chain ( $r$ ) and the final answer ( $a$ ).
Metrics: The study evaluates Alignment (agreement with ground truth), Pass Rate (proportion deemed correct), Overconfidence (incorrect answers judged correct), and Conservativeness (correct answers judged incorrect).

Controlled Experiments

To isolate the factors influencing judgment, the authors synthesized reasoning chains with manipulated attributes:

Fluency Disruption: Inserting factually correct but irrelevant common-sense statements into the reasoning chain to break coherence.
Factuality Degradation: Replacing the inserted statements with counterfactual (factually incorrect) information.
Positional Analysis: Injecting these statements at the beginning (prefix) vs. the end (suffix) of the reasoning chain.

3. Key Contributions

Systematic Investigation of Reasoning Exposure: The paper provides the first systematic analysis of how access to reasoning chains alters LLM judging behavior across factual and mathematical tasks.
Differentiation of Judge Capabilities: It establishes a clear dichotomy between "weak" and "strong" judges regarding their susceptibility to reasoning chains.
Disentanglement of Signals: Through controlled experiments, the authors identify fluency and factuality as the two critical signals driving judge decisions, rather than the logical validity of the reasoning itself.
Positional Sensitivity: The study reveals that judges are significantly more sensitive to errors appearing at the beginning of a reasoning chain compared to the end.

4. Key Results

Observation 1: Weak Judges are Easily Misled

Inflated Pass Rates: Weak judges (e.g., Qwen3-8B) show a dramatic increase in Pass Rate when reasoning chains are provided, even when the underlying answer is incorrect.
Mechanism: Weak models tend to over-trust the presence and fluency of reasoning. They often accept incorrect answers if the reasoning chain forms a locally coherent, plausible narrative, even if it starts from a false premise (e.g., Figure 1 shows a model accepting a wrong answer because the reasoning "flows" well).
Data: On NQ, weak judges' Pass Rates jumped from ~58% to ~88% when reasoning was visible, despite the ground truth accuracy being only ~23%.

Observation 2: Strong Judges are Selective but Still Vulnerable

Improved Selectivity: Strong judges (e.g., DeepSeek-V3.1, GPT-4o) often decrease their Pass Rate when seeing reasoning, indicating they can identify and reject errors within the chain. Their Overconfidence decreases significantly.
The "High-Quality" Trap: Even strong judges are not immune. When exposed to reasoning chains generated by top-tier models (like DeepSeek-V3.1), strong judges are frequently misled. They perceive the high-quality, structured reasoning as evidence of correctness, even when the answer is wrong.
Over-Skepticism: Strong judges sometimes exhibit excessive conservatism, incorrectly rejecting correct answers because they find minor flaws in the reasoning chain.

Observation 3: Fluency and Factuality are Critical Signals

Fluency: Disrupting the fluency of a reasoning chain (by inserting irrelevant but true facts) causes a sharp drop in Pass Rates for almost all models. This suggests judges rely heavily on the smoothness of the text.
Factuality: Introducing counterfactual (false) information reduces Pass Rates, but the effect is non-linear. The quantity of errors matters less than the salience of the error.
Positional Bias: Errors introduced at the beginning of a chain have a much stronger negative impact on the judge's decision than errors at the end. This mirrors human cognitive bias where early errors invalidate the entire argument.

Observation 4: Self-Judging Mirrors External Judging

Models judging their own outputs exhibit similar biases to when they judge others, confirming that the presence of the reasoning chain itself (regardless of the judge's identity) is the primary driver of the bias.

5. Significance and Implications

Evaluation Paradigm Shift: The findings suggest that current "LLM-as-a-Judge" paradigms are flawed when evaluating modern reasoning models. Simply exposing reasoning chains does not guarantee better evaluation; it often introduces new biases.
Robustness Requirement: Future judge designs must move beyond passive consumption of reasoning. They need to be trained to critically verify reasoning steps rather than being swayed by fluency or superficial coherence.
Double-Edged Sword: Reasoning chains act as a double-edged sword: they provide richer information for strong judges to detect errors but serve as a powerful "smokescreen" for weak judges (and even strong ones) to mask incorrect answers.
Limitations: The study is limited to text-based benchmarks and does not explore multimodal reasoning or judges that generate their own reasoning before verdicts.

Conclusion: The paper concludes that while reasoning chains offer potential for better evaluation, current LLM judges lack the robustness to distinguish between genuine evidential value and superficial fluency. There is an urgent need for evaluation protocols that can critically assess reasoning quality rather than being persuaded by it.