💬 NLP

LLM Pedagogical Behavior in AI Tutoring Interactions

This study introduces a validated five-level scaffolding scale to analyze over 14,000 LLM tutoring responses, revealing that AI models overwhelmingly provide direct assistance (explaining or solving) rather than guided support, a behavior pattern that influences student dialogue but offers limited predictive value for exam performance beyond prior achievement.

Original authors: Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet hum of a university computer lab, a new kind of tutoring has taken root, one that never sleeps, never tires, and never judges. Students increasingly turn to large language models, powerful computer programs trained on vast amounts of text, to help them solve homework problems and understand difficult concepts. These digital assistants can do many things, but in the context of learning, a fundamental tension exists. If a tutor gives too little help, a student might get stuck and give up. If the tutor gives too much help, the student might simply copy the answer without ever learning how to think through the problem themselves. This balance is the core of the challenge: finding the right amount of support to guide a learner without doing the work for them. Researchers have long studied how human teachers navigate this space, but it remains unclear how these new, automated tutors behave when students use them in real, unscripted learning situations.

A team of researchers at KAIST in South Korea set out to map this uncharted territory. They analyzed a massive collection of conversations between students and an artificial intelligence tutor during an introductory artificial intelligence course. The students were free to ask for anything, from a simple hint to a complete solution, and the computer responded based on its default programming, which was simply to be a helpful assistant. The researchers wanted to know exactly what kind of help the computer was actually giving. To measure this, they created a five-step scale to categorize every response the AI made. At the lowest end, the computer might offer no real help at all or simply ask the student to think harder. In the middle, it might point out a relevant concept or a specific error without solving the problem. At the highest end, it would explain a method in detail or provide the complete, ready-to-use solution to the student's specific task.

When the researchers examined more than 14,000 responses from over 200 students, a striking pattern emerged. The AI almost never chose the middle ground. More than 95 percent of the time, the computer was either explaining a concept in great detail or simply solving the problem for the student. Responses that merely prompted the student to think or offered a small hint were incredibly rare, making up less than five percent of all interactions. Whether a student asked for help with writing code, editing a report, or understanding a complex theory, the computer's default reaction was to provide substantial, direct assistance. It acted less like a coach who guides a player through a drill and more like a mechanic who fixes the car for them.

This tendency to provide direct answers had a clear ripple effect on how the students behaved next. When the computer gave a full solution, the student was likely to ask for another piece of code or a new task to solve. When the computer offered an explanation, the student was more likely to ask a follow-up question about the underlying concept. The researchers found that the level of help the computer gave reliably predicted what the student would say in the very next turn of the conversation. However, this immediate reaction did not translate into better grades later on. The study looked at how well students performed on three major exams, which they took without any access to the computer. The amount of help the student received during their study sessions did not predict their exam scores any better than their past grades or the simple fact that they were talking to the computer at all.

The findings suggest that while these general-purpose AI tools are incredibly eager to help, they are not naturally inclined to teach in a way that forces students to struggle productively. They default to being efficient problem-solvers rather than patient educators. This does not mean the technology is useless, but it does reveal a specific behavior: without explicit instructions to hold back and guide, these systems will almost always do the heavy lifting for the student. The researchers conclude that to change this dynamic, educators and developers must deliberately design these systems to offer less direct help, rather than hoping the technology will naturally find the right balance on its own. The data provides a clear baseline of how these tools currently behave, showing that the path to effective AI tutoring may require us to teach the machines how to teach, rather than just how to answer.

Technical Summary: LLM Pedagogical Behavior in AI Tutoring Interactions

Problem Statement

As students increasingly utilize Large Language Models (LLMs) as on-demand tutors for coursework and problem-solving, there is a critical lack of empirical understanding regarding the level of assistance these models provide in authentic learning interactions. While existing research has characterized student usage patterns (what students ask) and explored pedagogically designed tutors, the actual assistance delivered by general-purpose LLMs remains under-characterized. This gap is significant because the degree of direct assistance determines the learner's role in the task, directly impacting the "Assistance Dilemma" (Koedinger & Aleven, 2007)—the tension between providing too little help (impeding progress) and too much help (bypassing productive cognitive effort).

Methodology

The study operationalizes the dimension of assistance as scaffolding level and develops a measurement framework to analyze 14,637 LLM responses from 203 students in an introductory university AI course (StudyChat dataset).

1. Scaffolding Scale Development

The authors define a five-level ordinal scale ( $L_0$ to $L_4$ ) to classify LLM responses based on the amount of direct assistance provided:

$L_0$ (Minimal): No substantive task-relevant assistance (e.g., refusals, redirection).
$L_1$ (Prompting): Encourages reasoning or trial without directing toward a solution.
$L_2$ (Hinting): Identifies relevant concepts, functions, or error locations without providing a concrete solution.
$L_3$ (Explaining): Explains an approach or concept that the student must still apply.
$L_4$ (Solving): Provides a complete, task-specific solution usable directly.

2. Data and Classification

Dataset: The StudyChat dataset (McNichols et al., 2026) containing interactions with a GPT-4o-mini model using a generic system prompt ("You are a helpful assistant") without specific pedagogical instructions.
Automated Classification: Responses were classified using GPT-4o-mini at temperature 0, trained on the defined codebook.
Validation: A stratified sample of 300 responses was annotated by two human experts. The classifier achieved 75.3% accuracy and a weighted F1 of .751 against adjudicated human labels. Agreement between human annotators was high ( $\kappa = .901$ ). Most classification errors occurred at the $L_3/L_4$ boundary, with a tendency to under-detect "Solving" responses.

3. Analytical Approach

The study addresses three Research Questions (RQs):

RQ1: What is the distribution of scaffolding levels, and how do they vary by request type and assignment?
RQ2: Is scaffolding level associated with the student's subsequent conversational behavior (next dialogue act)?
RQ3: Does scaffolding level provide additional predictive information regarding subsequent exam performance, beyond prior achievement and dialogue behavior?

Statistical methods included Kruskal–Wallis tests for assignment differences, Chi-squared tests for association between scaffolding and subsequent dialogue acts, and regression analysis to evaluate the incremental predictive power of scaffolding on exam scores.

Key Results

1. Distribution of Assistance (RQ1)

The LLM operates predominantly at high levels of assistance.

Concentration: Over 95% of responses are classified as either Explaining ( $L_3$ , 40.3%) or Solving ( $L_4$ , 55.2%).
Rarity of Low Assistance: Prompting ( $L_1$ ), Hinting ( $L_2$ ), and Minimal ( $L_0$ ) responses collectively account for fewer than 5% of all interactions.
Variation: While assistance levels vary significantly across different assignments and request types (e.g., writing requests receive the highest mean scaffolding, $\bar{L}=3.72$ ), high-assistance responses remain dominant across all instructional contexts.

2. Association with Subsequent Interaction (RQ2)

Scaffolding level is systematically associated with the student's next conversational move.

Distinct Patterns: After Solving ( $L_4$ ) responses, students most frequently make another writing request (29.7%). After Explaining ( $L_3$ ) responses, conceptual questions are most common (40.5%). After Prompting ( $L_1$ ), students often provide additional context (34.0%).
Robustness: This association persists even when controlling for the broad category of the student's initial request, indicating that the scaffolding level itself influences the immediate flow of dialogue.

3. Predictive Power for Performance (RQ3)

Scaffolding level provides little additional predictive information regarding subsequent exam performance.

Model Comparison: Regression models predicting exam scores ( $e_1, e_2, e_3$ ) using prior achievement and dialogue act counts explained 12–30% of the variance. Adding the mean scaffolding level increased the explained variance by only 1–2 percentage points.
Statistical Significance: None of the model comparisons reached conventional statistical significance ( $p > .05$ ). The estimated coefficient for scaffolding was inconsistent across exams (changing direction for $e_3$ ).
Conclusion: Once prior achievement and observed dialogue behavior are accounted for, the specific level of scaffolding does not significantly predict exam outcomes in this dataset.

Significance and Claims

The paper establishes an empirical baseline for LLM assistance in authentic tutoring interactions. Its primary contributions are:

Behavioral Baseline: It demonstrates that without explicit pedagogical prompting, general-purpose LLMs default to high-directness assistance (Explaining and Solving), rarely engaging in lower-scaffolding strategies like prompting or hinting.
Measurement Framework: It introduces a validated five-level scaffolding scale that can be used to evaluate how alternative tutoring designs (e.g., Socratic prompts, multi-agent systems) change the assistance students receive.
Distinction Between Interaction and Outcome: The findings suggest that while scaffolding level influences immediate interactional patterns (what the student does next), it does not necessarily translate to measurable differences in subsequent exam performance in this specific context.

The authors emphasize that these findings do not identify an "optimal" assistance level but rather provide a reference point. They argue that evaluating LLM tutors requires distinguishing between two questions: (1) Behavioral: Does a pedagogical intervention change the assistance students receive? and (2) Educational: Does that change improve learning? The scaffolding measure addresses the first question directly, while the second remains an open empirical question requiring further investigation.

Limitations

The authors acknowledge four main limitations:

Context Specificity: Data comes from a single introductory AI course and one model configuration; results characterize this setting rather than all general-purpose LLMs.
Classification Granularity: Automated classification is imperfect, particularly at the $L_3/L_4$ boundary, and low-assistance levels are rare, limiting fine-grained analysis of $L_0$ – $L_2$ .
Observational Nature: The study cannot establish causality; prompt content, task difficulty, and student characteristics may influence both the response and subsequent behavior.
Assessment Scope: The analysis evaluates incremental prediction rather than causal learning effects, and the concentration of responses at high scaffolding levels limits the variation available to detect performance differences.