Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability

This study reveals that the impact of LLMs on critical thinking is fundamentally shaped by time constraints, where early or continuous access enhances performance under time pressure but impairs it when sufficient time is available, whereas delayed or no access yields the opposite effect.

The Gist

Imagine you are trying to solve a complex puzzle, like figuring out the best way to fix a leaky dam in your town. You have a stack of confusing reports, news articles, and expert opinions to read. Now, imagine you have a super-smart robot assistant (an AI) that can read these documents for you and summarize them instantly.

The big question is: When should you let the robot help you?

This paper investigates exactly that. The researchers set up a giant experiment with nearly 400 people to see how the timing of the AI's help and the amount of time you have to do the work changes your ability to think critically.

Here is the breakdown of their findings using simple analogies.

The Two Main Rules of the Game

The researchers changed two things in their experiment:

1. Time Availability: Did you have 10 minutes (a frantic rush) or 30 minutes (a relaxed, thoughtful pace)?
2. AI Access Timing: When could you talk to the robot?
   • Early: The robot helps you right at the start.
   • Continuous: The robot is with you the whole time.
   • Late: You work alone first, and the robot only helps at the end.
   • No AI: You do it all yourself.

The Big Surprise: The "Time Reversal"

The most important finding is that the AI acts like a double-edged sword. What helps you when you are in a hurry can actually hurt you when you have plenty of time, and vice versa.

Scenario A: The "Fire Drill" (Insufficient Time)

Imagine: You are in a burning building and need to find the exit map in 10 minutes. You are panicking.

• What happened: People who let the AI help them immediately (Early or Continuous access) did a much better job than those who tried to figure it out alone first.
• The Analogy: The AI acted like a flashlight in the dark. When you are rushed, you don't have time to wander around looking for the light switch. The AI gave you the map instantly, allowing you to make a quick, decent decision.
• The Result: AI helped performance.

Scenario B: The "Deep Dive" (Sufficient Time)

Imagine: You are in a library with 30 minutes to study a complex history topic. You have plenty of time to think.

• What happened: People who let the AI help them immediately actually did worse than those who worked alone first. The people who worked alone first and only asked the AI at the very end (Late access) produced the best, most balanced arguments.
• The Analogy: The AI acted like a tour guide who talks too much. If you let the tour guide start talking the moment you walk into the museum, you might just listen to them and stop looking at the art yourself. You miss the details, you don't form your own opinions, and you end up with a shallow understanding.
• The Result: AI hurt performance.

Why Did This Happen? (The "Anchor" Effect)

The researchers found that when you have enough time, starting with the AI creates a mental "anchor."

• The Anchor: Once the AI gives you an answer or a summary, your brain latches onto it. Even if you have 30 minutes to think, you stop exploring other possibilities because you feel like you already have the "right" answer. You stop reading the original documents deeply.
• The Independent Thinker: People who worked alone first built their own "mental map" of the problem. When they finally asked the AI for help at the end, they used it like a spell-checker or a second opinion. They could say, "I thought X, but the AI suggests Y. Oh, that's a good point I missed!" This made their final decision much stronger and less biased.

The "Memory" Cost

There was another interesting side effect. When people had plenty of time but used the AI from the start, they forgot the details of the documents they read.

• Analogy: It's like having someone else read a book to you while you sit in a comfortable chair. You get the gist of the story, but you don't remember the specific plot points or character names as well as if you had read the book yourself. The AI did the "heavy lifting" of memory, so your brain didn't bother to store the information.

What Does This Mean for Us?

The paper suggests that we shouldn't just ask, "Is AI good or bad?" Instead, we need to ask, "When is AI good?"

1. In a Crisis (Time Pressure): If you are drowning in work and need a quick answer, let the AI help you right away. It's a great tool for speed and scaffolding.
2. For Deep Learning (Plenty of Time): If you are trying to learn something new or make a complex, important decision, do it yourself first. Struggle with the problem, read the sources, and form your own opinion. Then, bring in the AI to check your work, find blind spots, or balance your perspective.

The Takeaway

Think of the AI not as a replacement for your brain, but as a tool.

• If you are running a race, a pair of running shoes (AI) helps you go faster immediately.
• If you are trying to learn how to run, you need to practice your own form first. If someone else runs for you the whole time, you never learn the technique, and you might even forget how to run on your own.

The best way to use AI for critical thinking is to let it be the assistant, not the driver, especially when you have the luxury of time to think.

Technical Summary

Here is a detailed technical summary of the paper "Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability" by Zhi, Kumar, and Lee.

1. Problem Statement

While Large Language Models (LLMs) are increasingly integrated into cognitive tasks, their impact on critical thinking remains ambiguous. Existing research often relies on self-reports or simple comparisons of "LLM vs. No LLM," failing to account for the temporal context in which these tools are used.

The authors identify two critical, interacting time constraints that are often studied in isolation but are crucial in real-world scenarios:

1. LLM Access Timing: When the user can access the LLM (Early, Continuous, Late, or None).
2. Time Availability: The total duration allotted for the task (Insufficient/Time Pressure vs. Sufficient).

The core research question is how these two dimensions interact to shape human-AI collaboration outcomes in critical thinking tasks, specifically whether LLMs augment or undermine reasoning depending on when they are introduced and how much time the user has.

2. Methodology

Experimental Design

The study employed a 4 × 2 between-subjects factorial design with 393 participants recruited from Prolific.

• Factor 1: LLM Access Timing (4 levels):
  • Early: Accessible only in the first third of the task time.
  • Continuous: Accessible throughout the entire task.
  • Late: Accessible only in the final third of the task time.
  • No LLM: No access to the model.
• Factor 2: Time Availability (2 levels):
  • Insufficient: 10 minutes (inducing time pressure).
  • Sufficient: 30 minutes (allowing for deliberation).

Task and Environment

• Task: A critical thinking performance assessment based on the iPAL (International Performance Assessment of Learning) framework. Participants acted as city council members deciding on a water contamination proposal.
• Materials: Participants reviewed a curated set of 7 documents with varying relevance, trustworthiness, and stances (pro, con, neutral).
• Interface: A custom web interface controlled LLM access dynamically. The LLM (GPT-4o) was restricted to the provided documents only, preventing external knowledge retrieval.
• Procedure: Participants completed the task, followed by post-task assessments measuring Recall, Evaluation, and Comprehension, and a self-assessment of critical thinking.

Dependent Variables & Metrics

Performance was measured across four dimensions:

1. Essay Score: The primary metric, calculated by counting valid arguments, source references, and trustworthiness evaluations, while penalizing fabrications and errors.
2. Myside Bias: The absolute difference between pro and con arguments (lower is better).
3. Recall: Free recall of document main ideas.
4. Evaluation: Accuracy in judging document relevance, trustworthiness, and stance.
5. Comprehension: Accuracy in judging factual vs. counterfactual statements.

Analysis

The authors used Analysis of Covariance (ANCOVA) to examine main effects and interactions, controlling for participant demographics and prior LLM usage. Essay grading was performed by an LLM (validated against human raters with high inter-rater reliability, ICC > 0.70).

3. Key Results

The study revealed a "Temporal Reversal": the effect of LLM access timing on performance flips depending on time availability.

A. Under Time Pressure (Insufficient Time)

• Early/Continuous Access: Significantly improved Essay performance compared to working independently.
  • Mechanism: LLMs provided immediate scaffolding (summarization, initial argument generation) that helped users overcome the time constraint.
  • Data: Early access yielded the highest scores ($M=3.80$), outperforming Late access ($M=1.86$).
• Late/No Access: Performance was lowest. The LLM arrived too late to assist with rapid argument generation.

B. Under Sufficient Time

• Early/Continuous Access: Impaired performance compared to working independently.
  • Mechanism: Early access acted as an "anchor," causing users to rely on initial LLM outputs and reducing deep engagement with source documents.
  • Data: Late access ($M=5.77$) and No LLM ($M=5.76$) significantly outperformed Early ($M=4.51$) and Continuous ($M=4.71$) access.
  • Recall: Early/Continuous access significantly reduced Recall scores, suggesting users failed to internalize source material when the LLM was available from the start.
• Late/No Access: Participants who worked independently first showed the highest performance.
  • Myside Bias: Late LLM access significantly reduced Myside Bias compared to No LLM access while maintaining argument quantity. Users used the LLM at the end to check for overlooked perspectives.

C. Interaction Effects

• Time Availability Benefit: Sufficient time significantly boosted performance for those working independently (Late/No LLM) but provided minimal benefit for those with Early/Continuous access. This suggests that early LLM access prevents users from leveraging extra time for deeper deliberation.
• Self-Report vs. Performance: Self-assessed critical thinking showed minimal variation across conditions, highlighting that users are unaware of the cognitive shifts caused by LLM timing.

4. Key Contributions

1. Empirical Evidence of Temporal Reversal: The study provides the first systematic evidence that LLMs can be beneficial under time pressure but detrimental when time is sufficient, depending entirely on when they are introduced.
2. Mechanism Identification: Through behavioral logs and textual overlap analysis, the authors identified that:
   • Early access leads to anchoring (users stop iterating on documents).
   • Late access facilitates perspective balancing (users use the LLM to challenge their own biases).
3. Performance vs. Self-Report: Demonstrated that self-reports fail to capture the nuanced negative impacts of LLMs on memory retention (Recall) and deep processing under sufficient time.
4. Design Framework: Proposed a 2x2 framework (Figure 1 & 9) for researchers and designers to evaluate human-AI collaboration, moving beyond binary "use vs. no use" comparisons.

5. Significance and Implications

Design Implications for LLMs

• For Sufficient Time: Systems should encourage independent work first. Features could include "nudges" asking for initial thoughts before revealing the LLM, or friction-based delays to prevent premature offloading.
• For Time Pressure: Systems can provide immediate scaffolding (summarization, navigation) but must include safeguards (e.g., prompts for source diversity) to prevent users from anchoring to a single perspective.
• Dynamic Adaptation: LLM applications should detect time constraints (e.g., via timers or user input) and adapt their assistance strategy—offering deep reasoning support only after the user has engaged with the material.

Research Implications

• Re-evaluating Prior Studies: Many previous studies on LLMs in education or creative writing may have been conducted under "Sufficient Time" conditions, potentially missing the benefits of LLMs under pressure or the harms of early access.
• Methodology: Future human-AI collaboration research must prioritize performance-based assessments over self-reports and explicitly manipulate time constraints to understand the full scope of AI's impact on cognition.

Educational and Professional Impact

• Skill Maintenance: Relying on LLMs from the start may degrade long-term cognitive capacities (memory retention, internalization). Educators should design workflows that require independent reasoning before AI assistance to preserve critical thinking skills.
• Bias Reduction: Late LLM access is a powerful tool for reducing confirmation bias (Myside Bias) in decision-making, provided the user has time to form an initial hypothesis.

In conclusion, the paper argues that time is a central variable in human-AI collaboration. The utility of an LLM is not inherent but contextual; it shifts from a performance booster under pressure to a cognitive crutch under ample time, fundamentally altering how humans process, remember, and reason with information.