Technical Summary: PEERMATHDIAL

Problem Statement

Collaborative Problem Solving (CPS) is a recognized core competency in K-12 education, yet existing educational dialogue datasets predominantly focus on vertical interactions (teacher-student or tutor-student) or constructed interactions. There is a significant scarcity of resources centered on authentic, horizontal peer-to-peer interactions in small-group settings. This gap limits the ability of researchers to study how students naturally coordinate, negotiate meaning, and solve problems together in real educational environments. Furthermore, existing frameworks for analyzing CPS discourse, such as the PISA framework, often rely on coarse-grained categories that fail to capture the fine-grained, colloquial nature of middle school student dialogue.

Methodology

Data Collection and Processing

The authors introduce PEERMATHDIAL, a dataset collected from a middle-school (grades 6–8) mathematics summer camp organized by George Mason University in 2025.

• Participants: 27 focal students (out of 38 total participants) who completed background surveys.
• Setting: 90-minute collaborative problem-solving blocks involving small teams of 3–4 students working on 2–3 open-ended, real-world-inspired mathematical tasks (e.g., pricing a cake, coin problems).
• Recording: Sessions were captured using Swivl cameras for whole-class video and individual voice recorders at each table to ensure high-quality audio for small-group interactions.
• Transcription: A two-level workflow was employed using Otter AI for automatic speech recognition, followed by manual review by four student researchers. This process involved correcting lexical errors, merging incorrectly split utterances, and manually identifying speakers to replace generic labels with student identities.
• Dataset Scope: The final corpus contains 55 unique dialogue sessions comprising 6,406 turns (average 116.5 turns per dialogue). Student utterances account for 71.7% of the discourse.

Corpus-Grounded Dialogue Act Taxonomy

To address the limitations of predefined frameworks, the authors developed a new dialogue act taxonomy using a bottom-up, LLM-assisted inductive procedure:

1. Induction: An LLM (GPT-5.4) processed batches of raw dialogues to summarize recurring interactional behaviors.
2. Consolidation: Candidate acts were consolidated and refined via prompting to ensure appropriate granularity, avoiding overly broad or specific labels.
3. Iterative Refinement: The LLM assigned acts to all student turns, generating a confidence score. Low-confidence turns were re-examined to identify potential missing act types, leading to an updated inventory.
4. Expert Validation: Two math education researchers with teaching experience evaluated the taxonomy and a sample of 100 turns (50 random, 50 low-confidence). The experts confirmed the taxonomy's coverage, finding only 11 misclassifications in the low-confidence set, all of which were assignment errors rather than missing categories.

The resulting taxonomy organizes interactions into six functional stages:

• Affect and Off-Task Talk: Emotional reactions and social comments.
• Communicating and Coordinating: Acknowledging, explaining reasoning, guiding participation, and presenting to external listeners.
• Exploring and Understanding: Clarification, identifying constraints, and orienting to the activity.
• Monitoring and Revising: Challenging errors, checking step meanings, verifying constraints, and revising methods.
• Planning and Executing: Computation, decomposing structure, proposing solutions/strategies.
• Representing and Formulating: Constructing representations, mapping values, proposing patterns, and reusing cases.

Key Results and Applications

The paper demonstrates the utility of PEERMATHDIAL through three specific applications:

1. Temporal Evolution of CPS

By analyzing the distribution of dialogue acts across six temporal segments of conversations, the study reveals a non-static progression:

• Early Stages: Dominated by problem framing, identifying constraints, and coordinating participation.
• Mid-Stages: Shift toward operational and diagnostic actions, such as carrying out computations, challenging errors, and checking step meanings.
• Late Stages: Focus on evaluation, revising methods, and checking results against constraints.
• Teacher Intervention Impact: Teacher interventions were found to reliably shift student behavior away from local interpretation (e.g., identifying constraints) toward forward-moving mathematical activity (e.g., mapping representations to values, reusing patterns).

2. Linking Student Traits to Behaviors

By aligning dialogue act distributions with self-reported survey data (covering confidence, collaboration, and leadership), the study identifies distinct behavioral patterns:

• Leadership: "Leader" types produce more revising and explaining/justifying, while "Listener" types contribute more through acknowledging and identifying constraints.
• Collaboration: "Collaborative" students engage more in asking for clarification and explaining reasoning, whereas "Independent" students focus more on identifying constraints.
• Confidence: "Confident" students exhibit higher rates of off-task/affective commentary, while "Anxious" students more frequently identify constraints and refine representations.
• Discrepancy: The analysis highlights that self-reported traits (e.g., "avoiding arguments") do not always linearly predict specific dialogue acts (e.g., challenging errors), suggesting that situated behavior differs from stated dispositions.

3. Evaluation of LLM Student Simulation

The authors evaluated five state-of-the-art LLMs (including GPT-5.4, Claude Sonnet 4.6, and Qwen3.5) on the task of predicting the next dialogue act given only the student's survey profile and dialogue context.

• Result: All models achieved a low exact match rate, ranging between 16.0% and 20.0%.
• Implication: Current LLMs struggle to reproduce authentic, fine-grained collaborative behaviors based solely on high-level persona attributes, indicating a need for deeper fine-tuning or specialized modeling of mathematical collaboration states.

Significance and Claims

The paper claims that PEERMATHDIAL is the first dataset of face-to-face peer CPS dialogues collected from authentic K-12 mathematics classrooms. Its primary contributions are:

1. Resource Provision: It fills a critical gap in educational resources by providing a corpus of authentic peer interactions, enabling fine-grained analysis of collaborative dynamics that was previously limited by a lack of data.
2. Methodological Innovation: It introduces a corpus-grounded, LLM-assisted approach to dialogue act discovery, offering a more granular and context-sensitive alternative to rigid, expert-designed frameworks like PISA.
3. Analytical Utility: The dataset serves as a testbed for understanding the temporal dynamics of collaboration, the relationship between student traits and observed behaviors, and the current limitations of LLMs in simulating realistic student interactions.

The authors maintain a modest stance, acknowledging limitations such as the dataset's origin from a single summer camp (limiting generalizability to broader K-12 contexts) and the lack of multimodal data (gestures, gaze). They position the dataset as a foundational resource for future NLP and education research, particularly for developing high-fidelity virtual student simulators.