Not Too Short, Not Too Long: How LLM Response Length Shapes People's Critical Thinking in Error Detection

This study reveals that while the correctness of LLM-generated reasoning is the primary driver of user accuracy in critical thinking tasks, medium-length explanations uniquely enhance users' ability to detect errors when the AI's reasoning is incorrect, suggesting that response length plays a nuanced role in shaping human critical evaluation.

The Gist

Imagine you are a detective trying to solve a mystery. You have a very confident, well-spoken assistant (an AI) who gives you a theory about what happened. The assistant writes a long report explaining their reasoning.

Your job isn't just to solve the mystery yourself; it's to decide: "Is my assistant's report actually correct, or did they make a mistake?"

This is exactly what the researchers in this paper studied. They wanted to know: Does the length of the AI's explanation change how good you are at spotting its mistakes?

Here is the story of their findings, broken down into simple concepts:

1. The Setup: The "Goldilocks" Experiment

The researchers gave 24 smart professionals a series of logic puzzles (like a high-level IQ test). For each puzzle, an AI gave them an answer and an explanation.

The researchers played a trick:

• Sometimes the AI was right.
• Sometimes the AI was wrong (they planted errors in the logic).
• The AI's explanations varied in length: Short (a quick summary), Medium (a decent paragraph), and Long (a detailed essay).

They wanted to see if people were better at catching the AI's errors when the explanation was short, long, or just right.

2. The Big Surprise: "More" Doesn't Mean "Better"

You might think that if an AI gives you a huge, detailed, 500-word explanation, you would be more likely to spot a mistake because there's more information to analyze. Or, you might think a short answer is easier to check.

The study found neither of those things.

Instead, they found a "Goldilocks Zone" (a reference to the fairy tale where the porridge is "just right").

• When the AI was WRONG:

  • Short explanations: People missed the error. They thought, "Oh, it's short, it must be a quick fact," and trusted it too easily.
  • Long explanations: People also missed the error. The AI sounded so confident and wrote so much that people got overwhelmed or intimidated by the "wall of text." They assumed, "It must be right because it's so detailed."
  • Medium explanations: This was the sweet spot. When the AI gave a moderate-length explanation, people were much better at spotting the lie. They had enough text to think critically, but not so much that they got lost or felt pressured to agree.

• When the AI was RIGHT:

  • People got the answer right regardless of the length. If the AI was telling the truth, the length of the story didn't matter much.

3. The "Fluency Trap"

The paper suggests a psychological phenomenon at play. When an AI writes a long, flowing, confident-sounding essay, it tricks our brains. It feels authoritative.

Think of it like a salesperson:

• If a salesperson gives you a one-sentence pitch, you might be skeptical.
• If they give you a 20-minute presentation with charts and graphs, you might be so impressed by the effort that you forget to check if the product actually works.
• The "Medium" pitch is just enough to let you think, "Wait, does this actually make sense?" without dazzling you into silence.

4. What This Means for the Future

The researchers conclude that we shouldn't just let AI talk as much as it wants.

• Don't assume longer is smarter. A long explanation doesn't guarantee a correct one; in fact, it might hide mistakes better.
• Design matters. If we are building AI tools for doctors, lawyers, or business leaders, we should probably limit the length of the AI's explanations by default. We want the AI to give us "just enough" reasoning to help us think, not so much that we stop thinking for ourselves.
• Clarity over volume. The most important thing isn't how many words the AI uses, but whether its logic is clear and consistent.

The Takeaway

If you are using an AI to help you make a decision, be wary of the "long-winded" assistant.

If the AI gives you a massive, detailed explanation, don't just nod along. That might be a sign it's trying to dazzle you. A medium-length explanation is often the best friend to your critical thinking—it gives you enough to work with, but leaves room for you to say, "Hold on, I think I see a hole in that logic."

In short: Don't let the AI talk too much, or you might forget to listen to your own brain.

Technical Summary

1. Problem Statement

As Large Language Models (LLMs) become ubiquitous in professional and educational decision-support systems, there is a critical gap in understanding how specific properties of LLM outputs influence human critical thinking. While prior research has focused on model hallucinations and bias, less is known about how response characteristics (specifically length and correctness) affect a user's ability to evaluate information.

The core problem addressed is whether the length of an LLM's explanation acts as a cognitive scaffold that aids error detection or a cognitive burden that leads to over-reliance and reduced scrutiny. The authors investigate if longer, more detailed explanations inadvertently increase trust in incorrect reasoning (the "fluency effect") or if there is an optimal "sweet spot" for explanation length that maximizes human critical engagement.

2. Methodology

The study employed a within-subjects experimental design to isolate the effects of LLM output length and correctness on human judgment.

• Participants: 24 professionals (46% male, 54% female; ages 24–65) from diverse sectors (finance, healthcare, tech, etc.) in the US and UK.
• Task: Participants evaluated 15 modified Watson–Glaser Critical Thinking Appraisal (WGCTA) items. These items assess five facets: inference, assumption recognition, deduction, interpretation, and argument evaluation.
• Stimuli Design:
  • For each item, an LLM (GPT-5) generated a step-by-step analysis and a final Yes/No conclusion.
  • Correctness Manipulation: 8 outputs were logically correct; 7 contained "planted" errors in the final conclusion (while keeping the reasoning steps intact).
  • Length Manipulation: Outputs varied in word count (42–150 words). Instead of arbitrary thresholds, the authors used a data-driven, quantile-based approach to categorize outputs into Short, Medium, and Long tertiles.
• Procedure: Participants viewed the LLM output and judged its correctness ("Is the LLM's response correct?"). They also provided brief written rationales.
• Analysis: The primary metric was participant accuracy (correctly identifying a correct LLM output or correctly rejecting an incorrect one). Data was analyzed using mixed-effects logistic regression with a random intercept for questions to account for item difficulty.

3. Key Contributions

• Empirical Evidence on Length vs. Critical Thinking: The study challenges the assumption that "more detail equals better reasoning." It demonstrates that response length is not a linear predictor of critical thinking performance.
• Identification of a "Sweet Spot": The research identifies that medium-length explanations yield the highest error detection rates, particularly when the LLM is incorrect.
• Interaction Effect Discovery: The paper reveals a significant interaction between output correctness and length. The length of the explanation matters most when the LLM is wrong; when the LLM is right, length has a negligible impact on accuracy.
• Design Implications for AI Interfaces: The findings suggest that defaulting to long, verbose explanations may be detrimental to user autonomy and critical evaluation, advocating for intentional constraints on output length.

4. Key Results

The mixed-effects logistic regression revealed two main findings:

A. The Dominance of Correctness:
The correctness of the LLM output was the strongest predictor of participant accuracy.

• When the LLM was correct, participant accuracy was high across all lengths (71%–81%).
• When the LLM was incorrect, participant accuracy dropped significantly overall, but varied drastically by length.

B. The Length Interaction (The "Sweet Spot"):
When the LLM output was incorrect, the length of the explanation significantly influenced the user's ability to detect the error:

• Short Incorrect Outputs: ~25% accuracy (Users often failed to detect the error).
• Long Incorrect Outputs: ~31% accuracy (Users were still prone to accepting the error).
• Medium Incorrect Outputs: ~54% accuracy (Users were significantly more likely to detect the error).

Interpretation:

• Short outputs may lack sufficient context for users to engage deeply, leading to superficial processing.
• Long outputs may induce cognitive overload or "fluency bias," where the sheer volume of text creates an illusion of authority, causing users to defer to the conclusion without scrutinizing the logic.
• Medium outputs appear to provide enough structure to engage the user's critical faculties without overwhelming them or inducing over-trust.

5. Significance and Future Directions

Significance:
This work shifts the focus from "how accurate the AI is" to "how the AI presents information." It highlights a critical design pitfall: verbosity does not equal transparency. In decision-support systems, providing overly long explanations for incorrect reasoning can actively hinder human error detection. The study suggests that LLM interfaces should prioritize calibrated expressions of certainty and structural clarity over raw word count.

Limitations & Future Work:

• Cognitive Load: The study did not independently measure reading time or cognitive load, as these are intrinsically coupled with length. Future work should isolate these variables.
• Generalizability: The study used a single model (GPT-5) and English-speaking participants. Future research should compare different models and languages to see if these patterns hold universally.
• Control Condition: A control group without LLM assistance would help isolate the specific impact of AI on baseline critical thinking skills.

Conclusion:
The paper concludes that LLM response length should be treated as an intentional design choice. To support human critical thinking, systems should avoid defaulting to maximum verbosity and instead aim for concise, medium-length explanations that encourage active evaluation rather than passive acceptance.