Seeing the Reasoning: How LLM Rationales Influence User Trust and Decision-Making in Factual Verification Tasks

This study reveals that in factual verification tasks, users' trust and decision-making are primarily driven by the correctness and certainty framing of LLM rationales rather than their presentation format, highlighting the dual potential of well-designed rationales to either support decision-making or miscalibrate trust.

The Gist

Imagine you are asking a very smart, but sometimes overconfident, robot for advice on whether a specific city is on the West Coast of the USA. The robot says, "No, Miami is on the East Coast."

In the past, the robot might have just given you the answer. But now, modern AI (Large Language Models) often adds a "thinking process" section. It says, "Here is how I figured that out..." and lists its steps.

This paper asks a simple but crucial question: Does seeing the robot's "thinking steps" make you trust it more, less, or just the same? And does it change how you make decisions?

The researchers ran an experiment where they acted like "AI chefs" serving up different types of reasoning "recipes" to 68 people. They mixed and matched three ingredients:

1. The Timing: Did the thinking steps appear immediately, after a delay, or only if you clicked a button to see them?
2. The Truth: Was the reasoning actually correct, or did the robot make a silly mistake in its logic?
3. The Confidence: Did the robot sound super sure ("I am 100% certain!"), unsure ("I'm not really sure..."), or neutral?

Here is what they found, explained with some everyday analogies:

1. The "Confidence" Trap (Certainty Framing)

The Finding: When the robot sounded very confident, people trusted it more and followed its advice, even if the reasoning was shaky. When the robot sounded unsure, people trusted it less, even if the answer was right.

The Analogy: Think of the AI like a tour guide.

• If the guide says, "I'm absolutely positive this is the right path," you are likely to follow them, even if they are actually leading you in circles.
• If the guide says, "I think this might be the path, but I'm not 100% sure," you hesitate, even if they happen to be right.
• The Lesson: Confidence is a powerful "sales pitch." It can trick you into trusting a bad guide, or make you doubt a good one.

2. The "Broken Logic" Penalty (Correctness)

The Finding: If the reasoning steps contained a factual error (e.g., "Miami is in California"), people's trust tanked. They stopped trusting the answer, even if the final answer ("No, Miami isn't on the West Coast") was actually correct.

The Analogy: Imagine a math teacher who gets the final answer right (42) but shows a calculation on the board that is completely wrong (2 + 2 = 5).

• You might think, "Wait, if their math is wrong, how can I trust the answer?"
• The paper found that seeing a "broken" logic step is like seeing a crack in the foundation of a house. It makes the whole structure feel unsafe, even if the roof is fine.

3. The "Delivery Method" Didn't Matter Much (Presentation Format)

The Finding: Whether the thinking steps appeared instantly, slowly, or only when asked, it didn't really change how much people trusted the AI.

The Analogy: It's like receiving a letter.

• Whether the letter arrives by drone, by horse, or by hand, the content of the letter matters way more than how it was delivered.
• People cared about what the AI was saying (is it true? is it confident?), not when they saw it.

4. How People Actually Used the Reasoning

The Finding: People didn't use the reasoning to blindly accept the answer. They used it as a spot-check tool. They wanted to "audit" the robot.

The Analogy: Think of the reasoning as a receipt at a grocery store.

• You don't look at the receipt to decide what to buy; you look at it to make sure you weren't overcharged.
• The participants said, "I read the steps to see if the AI was hiding something or if it made a mistake."
• They wanted the reasoning to be step-by-step (like a clear receipt) rather than a long, confusing paragraph. They also wanted the AI to admit when it wasn't sure, so they could decide how much to rely on it.

The Big Takeaway

The paper concludes that showing an AI's reasoning is a double-edged sword.

• The Good: It helps you check the work and calibrate your trust.
• The Bad: If the AI is confident but wrong, or if the logic is messy, it can trick you into trusting a bad answer or make you doubt a good one.

The Advice for Designers:
Don't just make the AI sound smart and confident. Make sure the "thinking steps" are:

1. Accurate: Don't show steps that contradict the answer.
2. Honest: If the AI isn't sure, say so.
3. Checkable: Break it down into small, easy-to-read steps so humans can spot errors easily.

In short: Trust the AI's honesty and accuracy more than its confidence.

Technical Summary

Here is a detailed technical summary of the paper "Seeing the Reasoning: How LLM Rationales Influence User Trust and Decision-Making in Factual Verification Tasks."

1. Problem Statement

Large Language Models (LLMs), particularly Large Reasoning Models (LRMs), increasingly display step-by-step reasoning rationales alongside their answers. While these rationales are designed to improve model performance (e.g., via Chain-of-Thought), their impact on user trust and decision-making remains under-explored.

Key gaps identified by the authors include:

• Unfaithfulness: Rationales can be fluent and persuasive yet factually incorrect or inconsistent with the final answer.
• Misleading Certainty: Models often accompany rationales with confidence indicators (certainty cues) that may not align with actual reliability.
• Risk of Miscalibration: There is a concern that persuasive but flawed reasoning, especially when paired with high certainty, may lead users to trust incorrect advice or abandon independent fact-checking.

The study aims to determine how the presentation format, correctness, and certainty framing of LLM rationales influence user trust, decision confidence, and advice adoption in factual verification tasks.

2. Methodology

The authors conducted an online experiment with a mixed-factor design involving 68 participants.

• Design Structure:
  • Between-subjects factor: Rationale Presentation Format (3 levels):
    1. Instant: Rationale appears immediately with the answer.
    2. Delayed: Rationale appears after a brief, length-proportional delay.
    3. On-Demand: Rationale is hidden by default and revealed via a "Show/Hide" button.
  • Within-subjects factors:
    1. Rationale Correctness: Correct vs. Incorrect (controlled logical/factual flaws).
    2. Certainty Cue: None vs. Certain (e.g., "I am confident") vs. Uncertain (e.g., "I am not fully sure").
• Procedure:
  • Participants completed 6 trials covering all combinations of Correctness and Certainty.
  • Tasks involved factual verification questions (e.g., "Is Miami on the West Coast?").
  • Measures:
    • Advice Adoption: Binary choice to accept/reject the LLM's answer.
    • Decision Confidence: 7-point Likert scale.
    • Trust in Information: 5-point Likert scale regarding the specific response.
    • Trust in LLM System: 5-point Likert scale regarding the system generally.
  • Data Analysis: Mixed ANOVAs for quantitative data; Inductive thematic analysis for open-ended qualitative responses.

3. Key Contributions

1. Empirical Evidence on Rationale Impact: The study isolates the specific effects of rationale correctness and certainty framing, demonstrating that these factors significantly outweigh presentation timing in influencing user behavior.
2. The "Calibration" Role of Rationales: It reframes user-facing rationales not merely as persuasive narratives but as trust calibration tools. Users actively audit reasoning steps to verify logic rather than accepting them as definitive proof.
3. Identification of Double-Edged Risks: The paper highlights that while rationales can support decision-making, they can also miscalibrate trust when weak reasoning is paired with strong certainty cues or when reasoning contradicts the final answer.
4. Design Guidelines: Provides concrete, evidence-based recommendations for designing LLM interfaces that support appropriate trust and verification.

4. Key Results

Quantitative Findings

• Correctness is Critical: Correct rationales significantly increased trust, decision confidence, and advice adoption compared to incorrect ones. Incorrect rationales caused a sharp drop in trust, often lower than the "no rationale" baseline.
• Certainty Framing Drives Behavior:
  • Certain cues increased trust and confidence significantly compared to uncertain or no cues.
  • Uncertain cues reduced trust and confidence, often performing worse than having no cue at all.
  • Crucially: Certainty framing shifted user decisions even when the rationale content remained the same, indicating that the tone of confidence acts as a powerful heuristic.
• Presentation Format: The timing of the rationale (Instant vs. Delayed vs. On-Demand) showed no significant effect on trust or decision-making. Users were more sensitive to the reliability of the content than how it was revealed.

Qualitative Findings (Thematic Analysis)

• Theme 1: Auditing for Calibration: Users treat rationales as a "logic map" to audit the model. They check for omitted context or logical gaps to decide whether to trust the output.
• Theme 2: The Penalty for Inconsistency: A mismatch between the reasoning process and the final answer (even if the answer is correct) causes significant distrust. Users view this as "unfaithful" reasoning.
• Theme 3: Need for Structure: Users prefer stepwise, decomposed reasoning over narrative fluency. They want to see discrete facts to pinpoint exactly where an error occurs.
• Theme 4: Metacognitive Signals: Users value explicit uncertainty signaling and self-correction. They prefer interfaces that allow them to control the depth of reasoning (summary first, expand on demand).

5. Significance and Implications

This work challenges the assumption that "more transparency" (showing reasoning) is always beneficial. Instead, it argues for calibrated transparency:

• Design Recommendation 1 (Verification Scaffolds): Rationales should be presented as stepwise, auditable units tied to evidence, allowing users to trace the logic rather than just reading a narrative.
• Design Recommendation 2 (Consistency Checks): Systems must ensure the rationale does not contradict the final answer. Inconsistency is disproportionately damaging to trust.
• Design Recommendation 3 (Certainty Calibration): Certainty cues should be strictly aligned with the actual reliability of the reasoning. Overconfident language for low-quality reasoning is a high-risk design pattern.
• Design Recommendation 4 (Adaptive Interfaces): Interfaces should offer progressive disclosure (summary first, details on demand) and allow users to adjust the depth of reasoning based on their confidence levels.

Conclusion: LLM rationales are a double-edged sword. When designed poorly (flawed reasoning + high certainty), they can mislead users. When designed well (stepwise, evidence-based, and honest about uncertainty), they serve as powerful tools for users to calibrate trust and make better-informed decisions.