🤖 AI

Chat-Based Support Alone May Not Be Enough: Comparing Conversational and Embedded LLM Feedback for Mathematical Proof Learning

In a study of 148 undergraduate students using the GPTutor system, findings indicate that while structured, embedded proof-review feedback did not harm learning outcomes, reliance on chatbot-based support—particularly answer-seeking behavior—was negatively associated with subsequent exam performance, suggesting that conversational AI alone may not effectively foster transfer to independent mathematical proof assessments.

Original authors: Eason Chen, Sophia Judicke, Kayla Beigh, Xinyi Tang, Isabel Wang, Nina Yuan, Zimo Xiao, Chuangji Li, Shizhuo Li, Reed Luttmer, Shreya Singh, Maria Yampolsky, Naman Parikh, Yvonne Zhao, Meiyi Chen, Sca

Published 2026-04-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eason Chen, Sophia Judicke, Kayla Beigh, Xinyi Tang, Isabel Wang, Nina Yuan, Zimo Xiao, Chuangji Li, Shizhuo Li, Reed Luttmer, Shreya Singh, Maria Yampolsky, Naman Parikh, Yvonne Zhao, Meiyi Chen, Scarlett Huang, Anishka Mohanty, Gregory Johnson, John Mackey, Jionghao Lin, Ken Koedinger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning how to write a complex, logical story (a mathematical proof). You have a new, super-smart AI assistant available to help you. But this assistant comes in two very different "personas":

The Chatty Friend (The Chatbot): You can text back and forth. You ask, "How do I start?" and it might give you hints, or if you push hard enough, it might just tell you the answer.
The Strict Editor (The Proof-Review Tool): You must write your story first. Then, you paste it in. The tool doesn't talk to you; it just highlights specific sentences in red and writes comments in the margins saying, "This logic doesn't follow," or "Great job here." It refuses to write the story for you.

The researchers at Carnegie Mellon University wanted to see which of these two helpers actually made students better at math. They gave 148 students access to a system called GPTutor that had both tools.

Here is what they found, broken down simply:

1. The "Homework vs. Exam" Surprise

When students had access to the AI, their homework scores went up. It was like having a safety net; they could draft their proofs, get feedback, fix mistakes, and turn in better work.

However, when it came time for the midterm exams (where they couldn't use the AI), the students who used the tool did not score higher than those who didn't.

The Analogy: It's like practicing basketball with a coach who holds the hoop for you. You make a lot of shots during practice (homework), but when you play a real game without the coach (the exam), you miss just as often as everyone else. The practice didn't translate to real skill.

2. The Danger of the "Chatty Friend"

The study dug deeper to see how students used the tools. They found a scary pattern with the Chatbot:

Students who felt less confident in their math skills (low "self-efficacy") used the chatbot the most.
Crucially, the more these students chatted with the bot, the worse they did on the exams.
The Metaphor: The chatbot was like a "crutch." Students with shaky legs leaned on it so much that their own muscles (their own reasoning skills) never got strong enough to walk on their own. Even though the bot was programmed to give hints, students often bypassed the hints and just asked for the answer, short-circuiting their own learning.

3. The Safety of the "Strict Editor"

The Proof-Review Tool told a different story.

Students who used this tool also tended to be the ones who were struggling.
But, using this tool did not hurt their exam scores. It didn't help them skyrocket, but it didn't drag them down either.
The Metaphor: This tool was like a "mirror." You had to stand in front of it and look at your own work. It pointed out your flaws, but you still had to fix them yourself. Because you were forced to do the heavy lifting, you didn't lose your ability to think independently.

The Big Takeaway

The paper's main message is: Just having a smart AI chatbot isn't enough to teach you hard things.

If you let an AI chat with you freely, you might get lazy and stop thinking for yourself. You might get good at using the tool, but you won't get good at the subject.

However, if you force the AI to act like a strict editor—where you have to do the work first, and the AI only critiques your specific mistakes—you avoid the trap of over-reliance. You still struggle (which is good for learning), but you don't outsource your brain to the machine.

In short: Don't let the AI write the story for you. Let it edit your story, but make sure you are the one holding the pen.

1. Problem Statement

The paper addresses the challenge of integrating Large Language Models (LLMs) into undergraduate discrete mathematics courses, specifically for proof writing. While LLMs offer scalable, on-demand support, there is a critical concern that conversational interfaces may encourage "overreliance," where students outsource reasoning and planning to the AI rather than engaging in active sense-making.

The core research questions are:

Does earlier access to an LLM-powered tutoring system improve both homework performance and independent exam scores?
Do different interface designs (conversational chatbot vs. structured proof-review) create distinct pathways linking learner characteristics (e.g., self-efficacy) to learning outcomes?

2. Methodology

System Design: GPTutor
The authors developed GPTutor, an LLM-powered system with two distinct components:

Proof-Review-GPTutor (Structured): A non-conversational interface where students must first submit a proof attempt. The system provides high-level feedback and localized, visual highlights on specific problematic segments. It uses in-context learning with reference solutions but prohibits open-ended dialogue to prevent answer extraction.
AI Chatbot (Conversational): A GPT-4o based chatbot constrained to mathematics. It uses Socratic questioning and partial hints. However, the interface allows free-form dialogue, and students can bypass scaffolding to seek direct solutions.

Study Design

Participants: 148 undergraduate students from a discrete mathematics course at a large US research university (after attrition from an initial 155).
Design: A staggered-access randomized controlled trial.
- Experimental Group (n=77): Granted access to GPTutor after Midterm 1.
- Control Group (n=78): Granted access only after Midterm 2.
Data Collection:
- Academic Performance: 10 homework assignments and 3 midterm exams.
- Interaction Logs: Timestamps and counts of usage for both components, aggregated by exam intervals.
- Self-Efficacy: A baseline survey administered after Midterm 1 (Cronbach's $\alpha$ = 0.91).
Constraints: Students were not permitted to use AI tools during exams. Course policies allowed other external resources.

Analysis Strategy

Mixed-effects regressions: To compare homework scores between groups, controlling for prior performance.
Ordinary Least Squares (OLS): To predict exam scores based on access conditions.
Serial Mediation Analysis: A model testing the pathway: Baseline Self-Efficacy $\rightarrow$ Midterm 2 Performance $\rightarrow$ Component Usage (Midterm 2–3 interval) $\rightarrow$ Midterm 3 Performance. Bootstrapping (5,000 resamples) was used to test indirect effects.

3. Key Results

A. Access Impact: Homework vs. Exams

Homework: Earlier access to GPTutor significantly improved homework scores (HW5–HW6) for the experimental group compared to the control group ( $\beta = 0.23$ , $p = .026$ ).
Exams: Despite the homework gains, no significant difference was found in Midterm 2 or Midterm 3 scores between the groups. The benefits of AI support did not transfer to independent assessment.

B. Mediation Analysis: Distinct Pathways
The serial mediation analysis revealed that the two components function differently regarding learner characteristics and outcomes:

Self-Efficacy & Usage: Lower self-efficacy predicted higher usage of both the chatbot and the proof-review tool.
Chatbot Pathway: Chatbot usage constituted a negative associative pathway to exam performance.
- Lower self-efficacy $\rightarrow$ Higher Chatbot Usage $\rightarrow$ Lower Midterm 3 Scores.
- The indirect effect was significant (Indirect = 0.69, 95% CI [0.14, 1.56]).
Proof-Review Pathway: Proof-review usage showed no detectable negative association with exam performance.
- The indirect effect was not significant (Indirect = -0.24, 95% CI [-0.72, 0.12]).
Direct Effects: Self-efficacy and prior exam performance (Midterm 2) remained strong direct predictors of Midterm 3 performance, even after accounting for tool usage.

4. Key Contributions

Interface Design Matters: The study provides empirical evidence that the type of LLM interface significantly impacts learning outcomes. Conversational support alone may hinder transfer to independent assessment, whereas structured, work-anchored feedback does not show negative associations.
The "Chatbot Risk" Hypothesis: The authors identify a specific risk pathway where conversational reliance acts as a mediator between low self-efficacy and poor exam performance, suggesting that open-ended dialogue may encourage students to bypass cognitive struggle.
Staggered-Access Evaluation: The study demonstrates a robust methodology for evaluating AI tools in real-world educational settings, distinguishing between immediate practice benefits (homework) and transferable learning (exams).
Behavioral Insights: The findings suggest that while struggling students seek help more frequently, the mode of help-seeking (chat vs. structured review) determines whether that help is productive or detrimental to long-term learning.

5. Significance and Implications

Pedagogical Shift: The paper argues that simply deploying LLMs as chatbots is insufficient for complex tasks like proof writing. Effective AI tutoring must anchor feedback in student work and constrain interactions to prevent "short-circuiting" reasoning.
Assessment of Transfer: The lack of transfer from homework to exams highlights a critical limitation of current LLM-assisted learning: students may become proficient at using the tool to complete assignments but fail to internalize the reasoning required for independent problem-solving.
Future Design Directions: The authors recommend moving away from generic chat interfaces toward structured, work-anchored feedback systems. They suggest integrating mechanisms that regulate reliance, such as requiring proof attempts before feedback or using animated pedagogical agents to guide generative processing rather than providing direct answers.
Intervention Targeting: The correlation between chatbot usage and lower performance suggests that usage logs could serve as early warning signals to identify students who are struggling and may need alternative scaffolding strategies.

Conclusion: The paper concludes that while LLMs can improve immediate task performance (homework), conversational support alone does not reliably support the transfer of learning to independent assessments. Structured, evidence-based feedback mechanisms are essential to ensure AI supports, rather than replaces, student reasoning.