Trust and Reliance on AI in Education: AI Literacy and Need for Cognition as Moderators

This study of 432 undergraduate students reveals that while higher trust in an AI assistant generally leads to less appropriate reliance on its suggestions during programming tasks, this negative relationship is significantly mitigated by students' AI literacy and need for cognition, highlighting the importance of these traits in fostering critical evaluation of AI assistance.

The Gist

Imagine you are learning to cook a complex new dish. You have a brand-new, incredibly confident sous-chef (the AI) standing next to you. This sous-chef is fast, speaks clearly, and sounds like an expert. Sometimes, they give you the perfect instruction. Other times, they confidently tell you to add salt to a dessert or to bake the cake at 500 degrees.

The big question this research paper asks is: How much do you trust this sous-chef, and does that trust make you a better or worse cook?

Here is the story of what the researchers found, broken down into simple concepts.

1. The Experiment: The "Trap" Kitchen

The researchers put 432 college students in a digital kitchen. They gave them 14 programming puzzles (like figuring out what a piece of code does). For every puzzle, a chatbot (the AI) gave them a hint and an explanation.

Here's the twist: The AI was a bit of a trickster.

• On 8 puzzles, the AI gave perfect advice.
• On 6 puzzles, the AI gave confidently wrong advice (like telling you to use a fork to eat soup).

The researchers watched to see if the students would:

• Listen when the AI was right (Good Reliance).
• Ignore the AI when it was wrong (Good Judgment).
• Blindly follow the AI even when it was wrong (Over-reliance).

2. The Big Surprise: Trust is a Double-Edged Sword

You might think, "If I trust my sous-chef, I'll do a better job." The study found the opposite.

The more the students trusted the AI, the worse they got at spotting its mistakes.

Think of it like a hypnotist. When you trust someone too much, you stop checking their work. You stop asking, "Wait, does that make sense?" and just nod along.

• Students with high trust tended to accept the AI's wrong answers as if they were right. They stopped thinking for themselves.
• Students with lower trust were more skeptical. They checked the AI's work, caught the errors, and did better overall.

The researchers found a "curved" relationship: As trust went up, the ability to tell right from wrong went down. It wasn't a straight line; the drop-off happened quickly once students started feeling too comfortable with the AI.

3. The "Superpowers" That Helped (Moderators)

The researchers wondered: "Is there anyone who can trust the AI and still stay smart?" They looked at two specific "superpowers" the students had:

A. AI Literacy (Knowing how the machine works)

Imagine a mechanic who knows how a car engine works versus someone who just thinks the car is magic.

• The Finding: Students who understood how AI works (AI Literacy) were better at spotting mistakes when they were skeptical.
• The Catch: Even these experts started to get lazy and stop checking the AI's work once their trust got too high. Knowing how the engine works didn't save them if they decided to just trust the mechanic blindly.

B. Need for Cognition (Loving to think hard)

Some people love solving puzzles and thinking deeply (high "Need for Cognition"). Others prefer to take the easy route.

• The Finding: People who love thinking were better at catching the AI's lies when they didn't trust it yet.
• The Catch: Just like the mechanics, once these deep thinkers started trusting the AI too much, they also stopped doing the hard work of checking the answers.

The Lesson: Being smart or knowing how AI works helps you stay safe, but only if you stay skeptical. If you trust the AI too much, your "superpowers" turn off.

4. Why This Matters for Schools

The study suggests that in a classroom, we can't just hand students an AI tool and say, "Go ahead, trust it."

If we don't teach students to verify the AI, they will likely become "cognitive zombies"—accepting whatever the machine says because it sounds confident.

The Solution?
The researchers suggest we need to build "guardrails" into learning:

• The "Think First" Rule: Make students write down their own answer before they are allowed to see what the AI says.
• The "Evidence" Check: Force students to explain why they agree or disagree with the AI.
• The "Skeptic" Mindset: Teach students that the AI is a helpful assistant, not an oracle. It's a tool to check your work, not a replacement for your brain.

Summary

In a world where AI is everywhere, trust is dangerous if it makes you stop thinking. The more you trust the AI, the more likely you are to believe its lies. The only way to stay safe is to keep your "critical thinking" muscles active, no matter how confident the AI sounds.

The Golden Rule: Trust the AI, but verify everything. Don't let the robot drive the car without you holding the steering wheel.

Technical Summary

1. Problem Statement

As generative AI (GenAI) systems are increasingly integrated into higher education, students frequently encounter AI-generated outputs during learning tasks. A critical challenge is appropriate reliance: the ability to distinguish between correct and incorrect AI suggestions.

• The Risk: Students may exhibit overreliance (accepting flawed AI output without verification) or underreliance (rejecting helpful AI output).
• The Gap: While prior research acknowledges the risk of "automation bias" and "cognitive offloading," there is limited empirical understanding of how trust in an AI assistant specifically influences reliance behaviors in educational settings. Furthermore, it is unclear how individual learner characteristics (e.g., AI literacy, cognitive style) moderate this relationship.
• Core Question: How does student trust relate to appropriate reliance during AI-assisted programming, and do learner characteristics (AI literacy, Need for Cognition, self-efficacy) moderate this relationship?

2. Methodology

Study Design & Participants

• Participants: 432 undergraduate students enrolled in computing-related courses at the University of Florida.
• Setting: A single-session laboratory experiment combining an AI-assisted programming task with pre- and post-task surveys.
• Task: Participants solved 14 Python output-prediction problems (varying difficulty: loops, conditionals, data structures).
• AI Interaction: A "Wizard of Oz" approach was used. An AI chatbot provided recommendations and reasoning for each problem.
  • Stimuli: 8 problems had accurate recommendations; 6 had intentionally misleading recommendations (containing logical errors or syntax mistakes).
  • Interaction: Participants could ask follow-up questions via an OpenAI API (gpt-3.5-turbo) integrated into the platform, though initial recommendations were pre-programmed to ensure consistency.

Operationalization of Variables

• Appropriate Reliance (Dependent Variable): A behavioral metric calculated as the proportion of trials where a student accepted a correct recommendation OR rejected an incorrect one.
  • Overreliance: Accepting a misleading recommendation.
  • Underreliance: Rejecting a correct recommendation.
• Trust (Independent Variable): Measured via a 4-item post-task survey (7-point Likert scale) assessing trust in the AI assistant.
• Moderators (Learner Characteristics): Measured via pre-task surveys:
  • AI Literacy: Understanding of AI systems and limitations.
  • Need for Cognition (NFC): Tendency to engage in and enjoy effortful cognitive activities.
  • Programming Self-Efficacy & Literacy: Confidence and perceived understanding of programming concepts.

Data Analysis

• Regression Models: Linear regression was used to predict appropriate reliance from trust.
• Non-linearity Check: A quadratic model was tested to see if the relationship followed a curvilinear pattern (e.g., an "optimal" mid-range trust).
• Moderation Analysis: Interaction terms were computed between standardized trust and each learner characteristic to determine if the slope of the trust-reliance relationship changed based on individual differences.

3. Key Results

Descriptive Statistics

• Reliance Behavior: Students showed high variability. On average, they appropriately relied on AI in 61.77% of opportunities.
• Bias Toward Acceptance: Overreliance was frequent (86.03% of misleading trials were accepted), while underreliance was rare (2.37% of correct trials were rejected).
• Trust Levels: Post-task trust was moderately high ($M = 5.01$).

RQ1: Trust and Appropriate Reliance

• Negative Correlation: There was a strong, significant negative linear relationship between trust and appropriate reliance ($r = -0.42, p < .001$).
  • Interpretation: Higher trust predicted lower appropriate reliance. Students who trusted the AI more were less likely to discriminate between correct and incorrect suggestions.
• Non-linearity: A quadratic model explained slightly more variance than a linear model ($R^2$ increased from .173 to .185). However, the curve did not reverse direction within the observed data range. This suggests that in this context, trust acts primarily as a driver of overacceptance rather than a mechanism for calibrated trust (where moderate trust might be optimal).

RQ2: Moderation by Learner Characteristics

Two significant moderators were identified: AI Literacy and Need for Cognition (NFC). Programming self-efficacy and programming literacy did not show significant interaction effects.

1. AI Literacy:

   • The interaction was negative ($b = -1.03$).
   • Effect: At low levels of trust, students with high AI literacy showed significantly higher appropriate reliance than those with low literacy. However, as trust increased, the advantage of high AI literacy diminished, and the negative slope of trust became steeper.
   • Implication: High literacy helps students be selective when they are skeptical, but high trust overrides this critical evaluation.

2. Need for Cognition (NFC):

   • The interaction was negative ($b = -0.99$).
   • Effect: Similar to AI literacy, students with high NFC demonstrated better discrimination (higher appropriate reliance) when trust was low. As trust increased, the benefit of high NFC in maintaining critical evaluation weakened.
   • Implication: Even students who naturally enjoy deep thinking are susceptible to automation bias when their trust in the system is high.

4. Key Contributions

1. Empirical Evidence of Trust-Reliance Decoupling: The study provides robust evidence that in educational GenAI contexts, higher trust does not lead to better calibration. Instead, it correlates with a failure to detect errors (overreliance), challenging the assumption that trust is inherently beneficial for human-AI collaboration in learning.
2. Identification of Moderators: The paper demonstrates that AI Literacy and Need for Cognition are critical protective factors, but only under conditions of low trust. Once trust becomes high, these individual differences lose their protective effect against overreliance.
3. Behavioral Operationalization: The study moves beyond self-reported reliance to measure actual behavioral reliance (accept/reject decisions) in a controlled environment with known ground truth (correct vs. misleading AI).
4. Contextual Nuance: The findings highlight that the "optimal trust" curve observed in traditional human-automation literature may not apply to GenAI in education, where the default tendency is toward uncritical acceptance (overreliance) rather than underreliance.

5. Significance and Implications

• Instructional Design: The findings suggest that simply increasing student trust in AI tools is not a goal; in fact, it may be counterproductive. Instructional strategies must explicitly teach verification routines.
• System Design (Cognitive Forcing Functions): To counteract the negative effects of high trust, systems should embed "cognitive forcing functions." Examples include:
  • Requiring students to commit to an answer before seeing the AI suggestion.
  • Prompting students to justify why they agree or disagree with the AI.
  • Forcing a verification step before final submission.
• AI Literacy Education: While AI literacy helps, it is not a silver bullet. It must be paired with instructional scaffolding that maintains a state of "healthy skepticism" even when the AI appears confident.
• Future Research: Future work should investigate how these reliance patterns hold up in longer, open-ended tasks (vs. short output prediction) and explore specific interventions (e.g., staged interfaces) that can sustain critical evaluation even as trust grows.

Conclusion

The paper concludes that while AI literacy and a high need for cognition can support selective engagement with AI, high trust acts as a dominant force that suppresses critical evaluation, leading to overreliance. Effective educational integration of GenAI requires moving beyond trust-building toward designing systems and curricula that enforce metacognitive monitoring and verification regardless of the student's level of trust.