Technical Summary: Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

Problem Statement

While Large Language Models (LLMs) are increasingly deployed to generate tutoring responses in mathematics, it remains unclear how closely their instructional behavior aligns with expert human practice. Prior research has established that LLMs exhibit systematic linguistic regularities distinct from human writing and that human tutors often outperform LLMs in identifying and correcting conceptual errors. However, existing literature largely treats linguistic differences and pedagogical effectiveness as separate domains. There is a lack of empirical work linking measurable linguistic characteristics of tutoring responses to pedagogical quality across both human and LLM tutors, particularly in the context of mistake remediation. This study addresses this gap by analyzing how instructional strategies and linguistic profiles differ between expert tutors, novice tutors, and LLMs, and how these factors correlate with perceived pedagogical quality.

Methodology

Data

The study utilizes a dataset from the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors. The dataset comprises 296 middle-school and elementary-level mathematics remediation dialogues. For each dialogue, responses were generated by:

• Human Tutors: One expert and one novice tutor.
• LLMs: Seven models of varying sizes and architectures, including GPT-4, Gemini, Sonnet (Anthropic), Mistral 7B, Llama-3.1-405B, Llama-3.1-8B, and Phi-3.
• Total Responses: 2,444 tutor responses.

Annotations and Features

Pedagogical Quality: Responses were annotated on four dimensions using a three-level scale (Yes, To some extent, No):

1. Mistake Identification
2. Mistake Location
3. Providing Guidance
4. Actionability
   These were mapped to numerical values (2, 1, 0) and summed to create a composite pedagogical quality score.

Instructional and Linguistic Features:

• Instructional Moves: Derived from accountable talk theory, three features were extracted using a transformer-based text-pair classification model (RoBERTa trained on the TalkMoves dataset):
  • Pressing for Accuracy: Challenging the correctness of the answer.
  • Pressing for Reasoning: Prompting explanation or justification.
  • Uptake (Restating/Revoicing): Reformulating the learner's response.
• Linguistic Features:
  • Response Length: Log-transformed token count.
  • Lexical Diversity: Measured via Measure of Textual Lexical Diversity (MTLD).
  • Readability: Flesch–Kincaid Reading Ease score.
  • Politeness: Probabilistic output from a transformer-based classifier detecting pragmatic markers.
  • Agency: Probabilistic output from a model estimating agentic expression (intention, control, action orientation).

Analytic Strategy

The analysis addressed three research questions:

1. RQ1 (Differences): Compared mean differences in instructional and linguistic features between LLMs, novice tutors, and expert tutors (baseline).
2. RQ2 (Quality): Assessed relative pedagogical quality by computing deviations from the turn-level mean across all tutors responding to the same student error.
3. RQ3 (Associations): Estimated a linear fixed-effects model at the conversation level to predict relative pedagogical quality from instructional and linguistic features. This approach controlled for unobserved contextual variables (e.g., task difficulty) by demeaning variables within each conversation.

Key Contributions

1. Characterization of Differences: The study systematically characterizes differences in instructional strategies and linguistic features across expert tutors, novice tutors, and a diverse set of LLMs responding to identical prompts.
2. Quality Comparison: It provides a comparative analysis of perceived pedagogical quality across these tutor groups, controlling for the instructional context.
3. Feature Association: It identifies specific instructional strategies and linguistic features that are positively or negatively associated with pedagogical quality, distinguishing between features that drive quality and those that do not.

Results

Instructional and Linguistic Profiles (RQ1)

• Instructional Moves: LLMs generally exhibited lower levels of "pressing for reasoning" and "pressing for accuracy" compared to expert tutors. Conversely, LLMs showed higher prevalence of "restating/revoicing" than both experts and novices.
• Length and Diversity: LLM responses were substantially longer and exhibited higher lexical diversity (MTLD) than expert tutors. Novice tutors produced shorter responses with lower lexical diversity.
• Readability: LLM responses scored significantly lower on readability metrics (Flesch–Kincaid) than expert tutors, indicating higher cognitive processing demands. Novice tutors were closer to experts in readability.
• Politeness and Agency: LLMs demonstrated higher politeness scores than experts. Agency patterns varied; some LLMs showed lower agency, while novices showed higher agency than experts.

Pedagogical Quality (RQ2)

• Expert vs. Novice: Expert tutors consistently achieved positive relative pedagogical quality scores, while novice tutors showed negative deviations.
• LLM Performance: LLMs spanned a wide range. Larger models (e.g., Llama-3.1-405B, Mistral, Claude Sonnet) approached the relative pedagogical quality of expert tutors on average. Smaller or lower-performing models (e.g., Phi-3) exhibited negative deviations similar to novice tutors.

Associations with Quality (RQ3)

Regression analysis revealed the following associations with perceived pedagogical quality:

• Positive Associations:
  • Pressing for accuracy ($\beta = 0.586$)
  • Lexical diversity (MTLD) ($\beta = 0.649$)
  • Restating/Revoicing ($\beta = 0.346$)
  • Pressing for reasoning ($\beta = 0.209$)
• Negative Associations:
  • Agency ($\beta = -0.693$)
  • Politeness ($\beta = -0.160$)
• No Significant Association:
  • Readability
  • Response length

Significance and Claims

The paper argues that pedagogical quality is best understood as a property of response-level features rather than tutor identity (human vs. AI). The findings suggest that effective feedback relies less on linguistic complexity or assertiveness and more on how linguistic choices align with instructional function.

Specifically, the study highlights that while larger LLMs can approach expert performance in overall quality, they exhibit systematic deviations in their instructional profiles. They tend to underuse critical discursive strategies (pressing for accuracy/reasoning) while overusing others (revoicing) and generating responses that are longer, more lexically diverse, and more polite. The negative association of politeness and agency with pedagogical quality suggests that overly accommodating or "sycophantic" language may obscure instructional signals and weaken corrective feedback.

The authors conclude that for LLM-based tutors, optimization should focus on reproducing specific feature combinations associated with higher pedagogical quality (e.g., pressing for reasoning, restating) rather than simply increasing linguistic expressiveness or mimicking human tutors. These insights offer a pathway to making LLMs more pedagogically aware and steerable.

Limitations

The authors acknowledge several limitations:

• The dataset, while sufficient for within-turn comparisons, may benefit from larger scales for finer-grained analysis.
• Findings reflect model behavior under specific prompt configurations and may not represent the upper bound of LLM capabilities.
• The feature set is selective; other aspects like discourse structure or mathematical specificity were not examined.
• The study is limited to English language interactions in a mathematics context.
• Pedagogical quality is assessed via structured annotations of single-turn responses, not downstream student learning outcomes.