Technical Summary: Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study

1. Problem Statement

While Vision Language Models (VLMs) are increasingly adopted as personalized tutoring agents in mathematics education, there is a critical lack of systematic evaluation regarding their ability to adapt to diverse learner profiles. Current VLMs primarily function as "solvers" prioritizing answer correctness rather than as "educational agents" that provide pedagogically transparent guidance aligned with a learner's Zone of Proximal Development.

Existing evaluation paradigms largely mirror text-only settings, focusing on visual question-answering accuracy while neglecting instructional reasoning quality and the capacity to adapt guidance based on learner context (e.g., prior knowledge, learning attitude, self-efficacy). Furthermore, when learners do not explicitly share detailed performance data, it remains unclear whether VLMs can infer learner profiles or generate truly appropriate, differentiated instructional responses. This paper addresses the gap in evaluating whether VLMs can demonstrate adaptive instructional capabilities in visually grounded mathematics tutoring tasks.

2. Methodology

2.1 Learner Model-Based Rubric

The authors propose a novel rubric grounded in the adaptive learning framework (Shute and Towle, 2018) and learner characteristics identified by Bernacki et al. (2021). The rubric formalizes adaptivity assessment into three primary dimensions:

1. Adaptivity: The core evaluative focus, subdivided into:
   • Cognitive Aspects: Current knowledge, content scope (aligned with Common Core State Standards), and difficulty level.
   • Motivational Aspects: Learning attitude and self-efficacy.
   • Complexity: Explanation of question meaning, provision of examples, additional practice, and step-by-step cues.
2. Correctness: Evaluates the mathematical validity of the final answer and the solution's reasoning (Q10–Q12).
3. Quality: Assesses generation quality, including hallucinations, logical coherence, faithfulness to instructions, and formatting (Q13–Q19).

2.2 Experimental Design

• Dataset: A stratified dataset of 10 representative mathematics questions (5 from Grade 4, 5 from Grade 8) drawn from the Trends in International Mathematics and Science Study (TIMSS) 2019 and 2023. 90% of items are visually grounded (requiring diagrammatic interpretation).
• Learner Profiles: Six synthetic profiles constructed from TIMSS statistics, varying by grade (4/8), performance tier (High, Intermediate, Low), learning attitude, and self-reported confidence.
• Context Conditions: Four test groups ($C_1$ to $C_4$) with monotonically increasing learner-specific information:
  • $C_1$: Math question only.
  • $C_2$: Question + User prompt (attitude/confidence).
  • $C_3$: Question + System prompt (grade objectives) + User prompt.
  • $C_4$: Question + System prompt + User prompt + Full learner profile (including TIMSS scores).
• Models Evaluated: Five state-of-the-art VLMs (GPT-5, GPT-o1, Gemini-2.5-Flash, Llama3.2-11B-VL, Qwen3-30B-VL).
• Annotation: Four domain experts annotated 600 model responses using the rubric. Inter-rater agreement was strong ($\kappa = 0.87$).

3. Key Results

3.1 Adaptivity Performance

• Model Variance: There are significant differences in adaptivity across models ($F(4, 705) = 41.81, p < .001$). Gemini-2.5-Flash and Qwen3-30B-VL achieved significantly higher adaptivity scores than GPT-5, GPT-o1, and Llama3.2-11B-VL.
• Learner Profile Insensitivity: Adaptivity scores showed no significant association with learner performance levels ($p = .18$). Models tended to default to a "one-size-fits-all" pedagogical style, failing to differentiate instructional strategies based on specific learner needs (e.g., high-performing vs. low-performing learners).
• Context Sensitivity: Adaptivity significantly improved with the availability of more contextual information ($F(3, 708) = 92.73, p < .001$). Group 1 (question only) scored significantly lower than Groups 2–4. However, even with full context, models struggled to consistently produce learner-model-based responses.

3.2 Correctness and Quality

• Correctness: Driven primarily by model differences ($p < .001$). Gemini-2.5-Flash and GPT-based models outperformed Llama3.2-11B-VL. Correctness was robust across learner profiles, suggesting models either reached a performance ceiling with basic prompts or ignored supplementary system prompts.
• Quality: Significant variance existed across models ($p < .001$). Llama3.2-11B-VL produced significantly higher error counts. Qwen3-30B-VL achieved the highest average adaptivity but suffered in correctness and quality due to overly long generations that were often truncated or incomplete.

3.3 Specific Observations

• Motivational Neglect: Models exhibited minimal adaptation to learner attitudes and self-efficacy. For instance, GPT-o1 addressed self-efficacy concerns in only 8% of applicable cases.
• Knowledge Misalignment: Providing learner background did not guarantee appropriate feedback. In some cases, models provided more detailed explanations to high-performing learners than to intermediate learners who required more scaffolding.
• Multimodal Failures: Geometry items posed significant challenges. Models frequently conflated option labels with numerical values (e.g., answering "6kg" instead of "D") or hallucinated information from diagrams, leading to low correctness scores despite plausible reasoning.

4. Key Contributions

1. Learner Model-Based Rubric: The proposal of a systematic evaluation framework that disentangles pedagogical intent (adaptivity) from technical accuracy (correctness/quality), grounded in cognitive, motivational, and complexity dimensions.
2. Synthetic Evaluation Dataset: Construction of a human-annotated dataset based on TIMSS items, incorporating diverse learner profiles and controlled contextual conditions to simulate realistic tutoring scenarios.
3. Empirical Analysis: First-hand analysis of five SOTA VLMs, revealing that while models show some sensitivity to context, their adaptive behavior remains limited and inconsistent, particularly regarding motivational factors and differentiated scaffolding.

5. Significance and Claims

The paper claims that while VLMs show promise as educational tools, they currently lack the nuanced scaffolding required for genuine adaptive learning. The study highlights a gap between generic tutoring responses and truly adaptive instruction that aligns with a learner's specific academic background and psychological state.

The authors position this work as a baseline evaluation. They explicitly state that a response classified as "adaptive" in this study does not necessarily represent a high-level, high-quality adaptive response. The significance lies in providing a principled tool to systematically compare adaptive capabilities and identifying that current models often revert to generic scripts when critical context is introduced or removed. The work underscores the need for evaluation methodologies that prioritize pedagogical alignment with learners to guide the development of more educationally grounded multimodal models.

6. Limitations

The authors acknowledge several limitations:

• The rubric questions for adaptivity are not exhaustive (e.g., missing affective and socio-cultural variables).
• The limited number of math items may result in insufficient coverage of pedagogical adaptivity.
• The study used simulated learner profiles rather than real students, potentially missing complex learning behaviors inherent in human-AI interaction.
• The rubric serves as a baseline; it does not evaluate the depth or sophistication of adaptive strategies beyond the defined criteria.