💻 computer science

A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems

This paper proposes and evaluates a multi-agent framework that iteratively generates, validates, and refines LLM-created personalized math problems to address common issues like unrealistic contexts and mathematical inconsistencies, demonstrating that a single refinement cycle significantly improves problem quality across criteria such as realism and authenticity.

Original authors: Fareya Ikram, Nischal Ashok Kumar, Junyang Lu, Hunter McNichols, Candace Walkington, Neil Heffernan, Andrew S. Lan

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fareya Ikram, Nischal Ashok Kumar, Junyang Lu, Hunter McNichols, Candace Walkington, Neil Heffernan, Andrew S. Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to make math homework less boring. You know that if you can turn a dry equation about "apples" into a problem about "basketball" or "TikTok," your students will actually want to solve it. This is called personalization.

In the past, teachers had to do this manually, which took forever. Now, we have AI (Large Language Models) that can write these custom problems instantly. But there's a catch: AI is like a very enthusiastic but slightly clumsy intern. It tries hard to make things interesting, but it often makes silly mistakes.

The Problem: The "Clumsy Intern"

If you ask an AI to write a math problem about basketball, it might say:

"A basketball player jumps 500 feet into the air to dunk the ball."

That's mathematically solvable, but it's unrealistic (nobody jumps that high). Or it might use words so complex that a 7th grader can't read them. Or, it might pick a topic like "Classical Music" for a student who hates it, making the problem feel inauthentic.

The paper proposes a solution: Don't just ask the AI to write the problem once. Instead, build a "Quality Control Team" of AI agents to check and fix the work.

The Solution: The "Multi-Agent Factory"

The authors created a system that works like a specialized factory assembly line. Instead of one robot doing everything, they have a team of four different "inspectors," each with a specific job:

The Math Police (Solvability Agent): Checks if the math actually works. Does the answer exist? Is the logic sound?
The Reality Check (Realism Agent): Asks, "Does this make sense in the real world?" (e.g., "No, a water bottle doesn't cost $100,000.")
The Reading Coach (Readability Agent): Ensures the words aren't too hard or too easy for the student's grade level.
The Culture Guru (Authenticity Agent): Checks if the topic actually fits the student's life. (e.g., "Is this reference to a 1990s band something a 12-year-old today would care about?")

How It Works: The "Generate-Check-Fix" Loop

Here is the process, explained as a game of "Hot Potato":

Generation: The "Writer Agent" creates a draft problem based on a student's interest (e.g., "Basketball").
The Inspection: The draft gets passed to the four inspectors.
- If the Reality Check says, "7 inches is too small for a basketball court," the ball is thrown back.
- If the Culture Guru says, "This reference is outdated," the ball is thrown back.
The Revision: A "Fixer Agent" takes the criticism and rewrites the problem.
Repeat: This happens in a loop until the problem passes all four inspectors.

The Three Ways to Organize the Team

The researchers tested three different ways to manage this team to see which one fixed problems best:

Strategy A: The "Group Hug" (Centralized Refinement)
All four inspectors shout their complaints at once into a single pile. The Fixer Agent tries to read the whole pile and fix everything in one go.
- Result: Good, but the Fixer gets overwhelmed and might miss small details.
Strategy B: The "Project Manager" (Centralized with Planning)
The inspectors shout their complaints, but a "Project Manager" agent organizes them into a checklist first. It says, "Fix the math first, then the realism, then the words."
- Result: This was the best at fixing Authenticity (making the topic feel real) because it allowed for big-picture changes.
Strategy C: The "Specialized Repair Shop" (Decentralized Refinement)
Each inspector has their own dedicated Fixer. If the Reality Check finds a problem, only the Reality Fixer works on it. Then it passes to the next station.
- Result: This was the fastest at fixing Realism and Readability because the Fixers didn't get distracted by other people's problems.

What They Found

The Big Wins: The biggest mistakes in the first draft were usually Realism (silly numbers) and Authenticity (bad topics). But, just one round of this "check and fix" process fixed most of these errors.
The Human Test: The researchers asked real humans to grade the AI's work.
- The AI was great at checking math and reading levels.
- The AI was bad at checking "Authenticity." Why? Because "what's cool" is subjective. An AI might think a reference to "Baseball" is fine, but a human might know that the specific way the innings were counted in the problem was weird. The AI needs human help for the "vibe check."

The Takeaway

This paper shows that we can't just trust AI to write personalized schoolwork on the first try. It's like asking a chef to cook a meal for a picky eater; you need a team of critics to taste it, point out the salt, and fix the presentation before it goes to the table.

By using a team of specialized AI agents, we can turn a clumsy, error-prone AI into a reliable tool that creates math problems that are not only correct but also actually fun and relevant for students.

1. Problem Statement

Context personalization—tailoring math problems to students' individual interests (e.g., sports, music, gaming)—has been shown to improve engagement and learning outcomes. However, relying on Large Language Models (LLMs) to generate these personalized problems at scale introduces significant quality issues. The authors identify four recurring failure modes in current LLM-generated content:

Mathematical Inconsistency: Invalid solutions, ambiguous setups, or inconsistent quantities.
Unrealism: Quantities or scenarios that violate real-world norms (e.g., a water bottle costing $100).
Lack of Authenticity: Contexts that fail to align with students' actual lived experiences or age-appropriate interests.
Poor Readability: Increased linguistic complexity that burdens students.

Existing iterative refinement methods (like Self-Refine or Reflexion) often rely on undifferentiated self-feedback, which may overlook specific domain constraints. Furthermore, there is a lack of research on how to effectively coordinate feedback from multiple distinct quality criteria into a coherent revision process.

2. Methodology

The authors propose a Multi-Agent Framework that formalizes personalization as an iterative Generate–Validate–Revise loop.

A. Workflow Components

Conversion Agent: Takes an original math problem ( $p$ ), a target interest topic ( $t$ ), and a target reading level ( $g$ ) to generate an initial candidate problem ( $\hat{p}$ ).
Four Specialized Validator Agents: Each agent performs a binary pass/fail check and provides diagnostic feedback on a specific criterion:
- Solvability Agent: Verifies mathematical consistency and solution validity.
- Realism Agent: Checks if quantities, units, and scenarios are plausible within the target topic.
- Readability Agent: Ensures vocabulary and sentence complexity match the grade level (using Flesch–Kincaid metrics).
- Authenticity Agent: Verifies if the context is age-appropriate and relatable to middle schoolers.
Refinement Agent(s): Uses validator feedback to revise the candidate problem while preserving the original mathematical structure.

B. Three Refinement Strategies

The study compares three strategies for coordinating validator feedback:

Centralized Refinement: Aggregates all feedback from failed validators into a single prompt for one refinement agent.
Centralized Refinement with Planning: Inserts a Planning Agent between feedback aggregation and revision. This agent converts unstructured feedback into a prioritized action plan (e.g., fixing math errors before stylistic changes) before the refinement agent executes the plan.
Decentralized Refinement: Pairs each validator with a dedicated refinement agent. Revisions are applied sequentially in a fixed priority order (Solvability $\to$ Realism $\to$ Authenticity $\to$ Readability) to prevent cascading errors and information overload.

3. Experimental Setup

Dataset: 600 math problems from the ASSISTments platform.
Topics: Personalized to a fixed set of 20 student interest topics (spanning sports, entertainment, media).
Baseline: A "Zero-Shot" approach where the Conversion Agent generates a problem without any validator feedback.
Metrics: Validator failure counts across up to 3 refinement iterations.
Human Evaluation: Three human annotators evaluated 45 problems to measure the reliability of the LLM validators against human judgment (using Cohen's $\kappa$ and Fleiss' $\kappa$ ).

4. Key Results

A. Performance of Refinement Strategies

General Trend: Authenticity and Realism are the most frequent failure modes in initial (Zero-Shot) generations. A single refinement iteration substantially reduces these failures.
Decentralized Refinement: Converges fastest on Realism and Readability. Because each agent focuses on a single criterion, it avoids the complexity of balancing conflicting constraints simultaneously.
Centralized with Planning: Converges fastest on Authenticity. The authors suggest authenticity is a "global" property requiring holistic coherence, which benefits from a structured, prioritized plan rather than sequential localized edits.
Solvability: All strategies rapidly converge to near-zero failure rates after one iteration, indicating modern LLMs are already highly capable of maintaining mathematical consistency.

B. Validator Reliability (Human Evaluation)

Realism & Readability: The LLM validators show fair agreement with human annotators (Cohen's $\kappa \approx 0.32$ and $0.28$ respectively). The LLMs reliably capture objective constraints.
Authenticity: The LLM validator shows slight agreement with humans (Cohen's $\kappa = 0.133$ $κ = 0.133$ ) and near-random accuracy ( $51.1\%$ $51.1%$ ).
- Reasoning: Authenticity is highly subjective and dependent on specific domain knowledge (e.g., pop culture trends or sports rules). For example, an LLM flagged "7.1 innings" as unrealistic (since innings are usually thirds), while human annotators without baseball knowledge passed it. This highlights a gap where LLMs may lack the specific cultural nuance required for true authenticity.

C. Failure Patterns

Topic Dependence: Media-related topics (Twitter, TikTok) frequently triggered Authenticity failures, while sports/gaming topics triggered Realism and Readability failures.
Curriculum Dependence: Measurement and proportion units were prone to Realism errors, while expression-solving units were prone to Readability errors.

5. Key Contributions

Framework Design: Proposes a novel multi-agent architecture specifically for math context personalization, moving beyond single-prompt generation to a structured validation loop.
Strategy Comparison: Systematically evaluates how different coordination mechanisms (Centralized vs. Decentralized vs. Planning) affect different quality dimensions, revealing that no single strategy is optimal for all criteria.
Empirical Insight: Demonstrates that while LLMs are strong at mathematical consistency, they struggle significantly with Realism and Authenticity, and that these failures can be mitigated through iterative refinement.
Reliability Analysis: Provides critical evidence that while LLM validators are reliable for objective criteria (math, readability), they struggle with subjective criteria (authenticity), necessitating human-in-the-loop verification for the latter.

6. Significance

This work addresses a critical bottleneck in AI-driven education: the gap between generating "interesting" content and generating pedagogically sound, authentic content.

Practical Application: The proposed framework offers a scalable pathway for teachers to use LLMs for personalized content creation without sacrificing mathematical rigor or realism.
Future Directions: The findings suggest that future systems must incorporate domain-specific knowledge (e.g., pop culture databases) to improve authenticity validation and should prioritize human evaluation for subjective criteria. It also highlights the need for adaptive agent orchestration that treats different quality criteria (local edits vs. global coherence) differently.