Mathematics Teachers Interactions with a Multi-Agent System for Personalized Problem Generation

1. Problem Statement

The study addresses the challenge of personalizing mathematics education for K-12 students using Large Language Models (LLMs). While LLMs can generate context-rich problems aligned with student interests (e.g., sports, music, video games) to boost engagement, direct student-facing deployment poses significant risks:

• Pedagogical & Safety Risks: LLMs may generate harmful, misleading, or pedagogically inappropriate content.
• Hallucinations: LLMs often produce mathematically inconsistent problems, incorrect units, or unsolvable scenarios.
• Authenticity Gaps: Generated contexts often lack "realism" or fail to match the specific, fine-grained interests of individual students, leading to disengagement.
• Teacher-Student Distance: Fully automated personalization may erode the teacher-student relationship and bypass the teacher's professional knowledge of student needs.

The core research question is: What are the affordances and limitations of a multi-agent, teacher-in-the-loop approach to generating personalized math tasks?

2. Methodology

System Architecture: Persona Problem Builder (PPB)

The authors developed a web-based platform (React frontend, Flask backend) integrated with the ASSISTments homework system. The system utilizes GPT-4o as the underlying engine and employs a Multi-Agent System (MAS) architecture to ensure quality control before the teacher finalizes the problem.

The Workflow:

1. Input: A teacher provides a base math problem ID and a desired interest topic (e.g., "baseball").
2. Generation: The LLM generates a personalized version of the problem.
3. Multi-Agent Evaluation: Four specialized AI agents evaluate the generated problem in parallel:
   • Authenticity Agent: Checks if the context is age-appropriate and relatable.
   • Realism Agent: Verifies that quantities, units, and scenarios are plausible (e.g., no negative time, realistic scores).
   • Readability Agent: Assesses vocabulary and sentence complexity against the target grade level.
   • Hallucination Agent: Checks for mathematical consistency (e.g., correct options in multiple-choice, solvable equations).
4. Refinement Loop: If an agent fails, it returns specific issues to a Refinement Agent, which revises the problem. This iterates up to five times or until all agents pass.
5. Teacher-in-the-Loop: The teacher reviews the refined problem, makes manual edits (authoring), and submits it to the students.

Study Participants & Data Collection

• Participants: 8 middle school mathematics teachers (7th grade) and 364 students (207 with consent for demographic data).
• Task: Teachers used the PPB to create 212 personalized problems based on the Illustrative Mathematics (IM) curriculum, targeting three different interest topics per problem.
• Data Sources:
  1. Teacher Logs: 212 generation requests and 179 teacher refinement prompts (epistemic moves).
  2. Agent Logs: Binary pass/fail decisions and specific error reports from the four agents.
  3. Student Feedback: 237 open-ended responses regarding problem interest, improvement suggestions, and factual accuracy.
  4. Readability Metrics: Analysis using Coh-Metrix comparing original vs. personalized problems.

3. Key Contributions

1. Multi-Agent, Teacher-in-the-Loop Architecture: A novel system design that balances LLM automation with human oversight, specifically tailored for educational context personalization.
2. Empirical Evidence on Personalization Dimensions: Identification of which aspects of personalization are most difficult to achieve (specifically authenticity vs. realism).
3. Design Implications: Insights into balancing teacher control with student-level authenticity, highlighting that "fine-grained" personalization is difficult to automate without direct student input.

4. Results

A. Authenticity (The Primary Challenge)

• Teacher Actions: Teachers frequently modified topics to fit local contexts or specific student names (e.g., changing "pop" to "hiphop" or inserting specific student names).
• Student Feedback: This was the most critical friction point.
  • 83/235 positive comments cited specific likes (e.g., Taylor Swift, Call of Duty).
  • 160/235 negative comments cited dislikes or irrelevance (e.g., "I hate Taylor Swift," "Call of Duty isn't MY thing").
  • Key Finding: Authenticity is highly subjective. A "broad-grain" approach (one topic for a class) fails to satisfy individual students. Some students explicitly requested a "student-in-the-loop" approach where they could input their own interests directly.

B. Realism

• Agent Performance: The Realism Agent was highly effective. Teachers rarely flagged realism issues in the final versions, suggesting the agents successfully caught implausible quantities (e.g., "7.5 boxes of mac and cheese").
• Teacher Interventions: Teachers made 30 moves related to realism (adjusting units, quantities, or game mechanics like Roblox currency), indicating that while the agents caught many errors, teachers provided necessary domain-specific nuance.

C. Readability

• Metrics: The LLM successfully adjusted the Flesch-Kincaid grade level from ~4.4 (original) to ~6.6 (personalized), aligning better with 7th-grade standards.
• Teacher/Student Feedback:
  • Teachers rarely requested readability changes (only 7 prompts).
  • Paradox: While some students wanted simpler text, 10 students explicitly requested more detail and storytelling, suggesting that for engagement, students sometimes prefer higher cognitive load in exchange for richer context.

D. Mathematical Hallucinations

• Error Rates: Actual mathematical errors in the final problems were rare.
• Teacher Role: Teachers made 25 moves to clarify mathematical vocabulary or story logic (e.g., changing variable names, ensuring the story matched the math concept). This demonstrates that teachers use their professional knowledge to refine the framing of the math, even when the math itself is technically correct.

5. Significance and Implications

• The "Teacher-in-the-Loop" is Essential: The study confirms that while agents can handle technical constraints (readability, basic realism), they struggle with the nuance of student interest. Teachers act as the critical filter for "fit," using their deep knowledge of their specific classroom to curate content that AI cannot generate autonomously.
• Granularity vs. Scalability: There is a tension between the scalability of AI and the need for fine-grained personalization. Students expect problems tailored to their specific interests, which is difficult to achieve with a single topic prompt for a whole class.
• Future Directions:
  • Dynamic Agent Weights: The authors are moving toward systems where teachers can adjust the "weight" of agents (e.g., prioritizing realism over reading level).
  • Hybrid Models: Future systems may need to allow students to input their own interests directly, with the teacher acting as a curator rather than the sole prompt engineer.
  • Safety & Context: Removing the teacher entirely requires significant investment in safety and contextual fit, which current LLMs cannot yet guarantee without risking student disengagement or pedagogical harm.

In conclusion, the paper argues that multi-agent systems are powerful tools for augmenting teacher capabilities in personalization, but they cannot yet replace the teacher's role in ensuring the deep authenticity and relevance required to truly engage middle school students.