Technical Summary: GeoDial

Problem Statement

While visual information is critical for learning in domains like mathematics, particularly geometry, existing educational tutoring datasets are predominantly text-only. This limitation hinders the development of AI tutors capable of emulating human instructors who rely on diagrams and visual cues. Previous computer-based tutors (e.g., AutoTutor, Cognitive Tutor) utilized diagrams but faced scaling issues in authoring scripts. Although Large Language Models (LLMs) have addressed scaling in text-based tutoring, their initial iterations lacked visual capabilities. With the advent of Vision-Language Models (VLMs), there is an opportunity to reintegrate the visual modality into tutoring. However, there is a lack of datasets that capture how teachers coordinate language with visual guidance, such as diagram highlighting, to address student errors.

Methodology

Data Collection and Annotation Protocol

The authors introduce GeoDial, a multimodal tutoring dataset comprising over 1,300 teacher-student dialogs focused on geometry problem-solving. The data collection process involves a controlled setup where human math teachers interact with VLMs acting as students.

• Setup: Teachers are presented with a geometry problem, its diagram, the correct solution, and a student's incorrect solution (generated by a VLM). The conversation proceeds turn-by-turn until the teacher decides to end it.
• Annotation Interface: For each teacher turn, annotators must:
  1. Select Dialog Acts: Choose from a hierarchy of 5 acts (Generic, Focus, Probing, Telling, SeeFigure) and 16 subacts (e.g., AskLength, GetRelation, SeeHighlight) to define the pedagogical strategy.
  2. Highlight Diagrams: Use a digital pen tool to draw on the diagram. Highlights are color-coded (green for general, brown for secondary relations) and must correspond to the selected act (e.g., SeeFigure requires a highlight).
  3. Select Feedback: Choose positive, negative, or neutral feedback to prefix the utterance.
  4. Generate Utterance: Teachers select from or edit candidate utterances suggested by three VLMs (Gemini-Flash-2.5, GPT-4o, Claude-Sonnet-4) based on the selected strategy and highlights.
• Post-Processing: To reduce noise and standardize evaluation, free-form human highlights are converted into structured textual annotations representing lines, angles, arcs, and labels defined by named points. This allows for evaluation by models that cannot generate images directly.

Experimental Setup

The authors fine-tuned several VLMs and text-only models on GeoDial to evaluate their ability to generate the next teacher turn, including the dialog act, subact, feedback type, diagram highlights, and the textual utterance.

• Models: Evaluated models include closed-source VLMs (Gemini, GPT-4o, Claude), open-source text-only models (Qwen3, Llama 3.1), and open-source VLMs (Qwen3-VL, InternVL, Llama 3.2-Vision).
• Metrics:
  • Categorical: Macro-F1 for Acts, Subacts, and Feedback.
  • Utterance: BLEU and BERTScore.
  • Highlights: Due to the high frequency of empty sets (no highlight needed), the authors propose two metrics: Prediction Accuracy (predicting whether a highlight is needed) and Decision F1 (F1 score calculated only on instances where a highlight is required). A Combined Score is also defined to aggregate these.

Key Contributions

1. GeoDial Dataset: The first dataset of geometry tutoring dialogs that tightly integrates conversational teaching strategies with fine-grained, diagram-grounded visual highlights. It contains 1,341 dialogs with 5,168 annotated highlight instances.
2. Annotation Protocol: A scalable protocol that captures the coordination of language and visual actions, including a specific taxonomy for pedagogical acts and a mechanism for standardizing visual highlights into textual representations.
3. Empirical Analysis of VLMs: A comprehensive evaluation showing that while supervised fine-tuning (SFT) significantly improves the quality of generated tutor utterances and pedagogical scaffolding, current VLMs struggle to produce accurate diagram highlights.

Results

• Text Generation: Fine-tuned models substantially outperform base models and random baselines in generating pedagogically appropriate utterances, acts, and subacts. Vision-enabled models generally perform better than text-only models, with larger models (e.g., Qwen3-VL-32B) achieving the best overall results.
• Highlight Generation: Despite improvements in prediction accuracy (identifying when to highlight), models struggle with the Decision F1 (identifying what to highlight).
  • Fine-tuning often leads models to "abstain" from highlighting more frequently to avoid false positives, which lowers the Decision F1 score due to the high imbalance of empty-set examples in the data.
  • Focused experiments separating highlight prediction from utterance generation and using weighted training (oversampling positive examples) improved performance but did not fully resolve the difficulty.
  • Models trained on ground-truth utterances performed better at highlighting than those using model-generated utterances, suggesting a tight coupling between the text and the visual action that is difficult to learn simultaneously.

Significance and Claims

The paper claims that GeoDial exposes a distinct challenge for multimodal pedagogical modeling: the effective integration of visual reasoning with instructional interaction. While current methods can learn to generate high-quality tutoring dialogue, they fail to replicate the precise visual grounding (diagram highlighting) used by human instructors.

The authors position GeoDial as a resource to drive future research toward models that can better coordinate instructional language with visual guidance. They acknowledge limitations, including the use of simulated students (which may not perfectly capture all human student behaviors) and the restriction to geometry (a subset of mathematics). The work emphasizes that bridging the gap between text-based tutoring and visually grounded tutoring remains a significant hurdle for AI in education.