💬 NLP

MathDebugger: Detecting and Diagnosing Errors in Synthetic Mathematical Data

This paper introduces MathDebugger, a comprehensive benchmark comprising 6,000 manually verified mathematical instances with fine-grained error annotations, to evaluate and reveal the limitations of current large language models in detecting and diagnosing errors in synthetic mathematical data while demonstrating the value of explicit error-type information for improving data quality.

Original authors: Hao Liang, Meiyi Qiang, Yuying Li, Zefeng He, Xiaochen Ma, Ruichuan An, Yongzhen Guo, Zhengzhou Zhu, Bin Cui, Wentao Zhang

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Liang, Meiyi Qiang, Yuying Li, Zefeng He, Xiaochen Ma, Ruichuan An, Yongzhen Guo, Zhengzhou Zhu, Bin Cui, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot how to solve math problems. You don't have enough real homework for it to practice on, so you ask a computer program to invent thousands of new math questions and answers for it to study. This is called "synthetic data." It's like a chef creating a massive library of recipes to train a new cook. But here's the catch: the computer making the recipes might accidentally write a dish that requires "half an egg" or a math problem where the numbers don't add up. If the robot learns from these broken recipes, it will learn to be wrong.

For a long time, scientists have built tests to see if robots can solve math problems. But this paper asks a different, harder question: Can the robot spot the mistakes in the recipes before it even tries to cook them? Can it look at a question and say, "Wait, this problem is impossible," or look at an answer and say, "You calculated that wrong"? This is the difference between being a student who gets an A on a test and being a teacher who can grade the test and find the errors in the questions themselves.

The Math Detective's New Playground

Meet MathDebugger, a new playground built by researchers to test if our smartest AI robots can act as math detectives. Think of it as a giant "spot the difference" game, but instead of finding the extra dot in a picture, the AI has to find broken logic, missing information, or silly mistakes in math problems and their solutions.

The researchers built a massive dataset containing 6,000 items in total. They created 2,000 perfect math questions, 2,000 broken questions, and 2,000 annotated answers (where about 610 of them had mistakes). They didn't just make them broken; they made them broken in specific, tricky ways.

For the questions, they invented four types of traps:

Expression Errors: The sentence is a mess, like a recipe written in gibberish or with confusing grammar.
Lack of Conditions: The question asks for a solution but forgets to give a crucial number, like asking "How long is the trip?" without saying how fast you are driving.
Contradictions: The question gives two rules that fight each other, like saying "The box is empty" and "The box is full" in the same breath.
Unrealistic: The answer defies common sense, like a math problem where a person skips 1.5 times (you can't skip half a time in real life!).

For the answers, they created three types of errors: Logic (the reasoning steps are wrong), Computing (the math calculation is wrong), and Expression (the explanation is messy).

The Great AI Showdown

The researchers then invited 14 of the world's smartest AI models to play this game. This included famous closed-source giants (like GPT-o3 and Claude-3.5) and powerful open-source models (like LLaMA and Qwen). They also tested three specialized "Process Reward Models," which are AIs trained specifically to check the steps of a solution.

Here is the big surprise: Even the smartest robots are still terrible at this.

Despite being able to solve incredibly hard math problems, these AIs struggled to detect when a problem was broken.

On the "Question Detection" task (finding broken questions), the best models only got about 76% right on easy questions and 78% on hard ones.
On the "Answer Detection" task, they did a bit better, but when it came to identifying exactly what kind of mistake was made (the "Error-Type Classification"), the scores dropped significantly. The best model only got about 55% right on the hardest category.

The paper suggests that there is a "Solving-vs-Verifying Gap." It seems that AIs are great at doing the math (solving) but are surprisingly bad at checking the math (verifying). It's like a student who can ace a calculus exam but fails to notice that the teacher wrote the wrong formula on the board. Interestingly, the models specifically tuned for "reasoning" (like DeepSeek-R1) didn't necessarily do better at spotting errors than their general-purpose cousins; in fact, sometimes they did worse.

Why This Matters (and How to Fix It)

The researchers didn't just stop at saying "AI is bad at this." They asked: "What if we tell the AI what kind of mistake to look for?"

They tested a new strategy: giving the AI a hint about the error type before asking it to fix the problem. For example, telling the AI, "This question has a 'Contradiction' error," and then asking it to rewrite the question.

The results were a clear win. When the AI knew the error type, its ability to fix the problem improved significantly.
For "Lack of Conditions" errors, the improvement was nearly 5 percentage points, a statistically significant jump.
This proves that these error labels aren't just for grading; they are like a flashlight that helps the AI see exactly where to shine its correction efforts.

The Bottom Line

MathDebugger is the first tool that treats math data like a crime scene, looking for specific types of "crimes" (errors) rather than just asking "Is this right or wrong?" The paper shows that while our current AI models are incredibly powerful, they are not yet perfect editors. They can solve the puzzle, but they often miss the fact that the puzzle pieces don't fit together in the first place.

The researchers conclude that we need to keep building better tools to audit our training data. If we want AI to be truly reliable, it needs to learn not just how to solve math, but how to be a critical, type-aware detective that spots the broken questions before they ever reach a student's desk.

Technical Summary: MathDebugger

Problem Statement

Synthetic mathematical data is increasingly utilized to scale the training of Large Language Models (LLMs), yet the correctness of this generated data is not guaranteed. While prior work has focused on scoring reasoning steps (Process Reward Models) or detecting errors in multimodal contexts, there is a lack of benchmarks specifically designed to audit the textual correctness of synthetic mathematical question-answer (QA) pairs themselves. Existing tools often fail to identify errors inherent in the problem statement (e.g., contradictory conditions) or lack fine-grained taxonomies to classify specific error types, which is crucial for improving data synthesis pipelines.

Methodology

The authors introduce MathDebugger, a text-only, type-aware benchmark designed to jointly detect and classify errors in synthetic mathematical questions and answers.

Benchmark Construction

Data Sources: The dataset is built upon seed data from GSM8K (labeled SIMPLE) and MATH (labeled CHALLENGING).
Synthesis Strategy:
- Questions: The authors designed 10 novel question-rewriting prompts to generate erroneous questions and 16 rewriting prompts (an expansion over prior work like MuggleMath) to enhance diversity. These were generated using models such as Qwen2.5-72B-Instruct, LLaMA-3.1-70B-Instruct, and GPT-4o.
- Answers: Answers were regenerated using models of varying capabilities (Qwen2.5-72B, LLaMA-3.1-8B, Qwen2.5-Math-1.5B) with both Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting.
Error Taxonomy:
- Question Errors (4 types): Expression Error (grammar/ambiguity), Lack of Conditions, Contradictions, and Unrealistic (violating common sense/real-world constraints).
- Answer Errors (3 types): Expression, Logic, and Computing errors.
Annotation Protocol: A pool of 30 STEM graduate annotators and 10 reviewers from top-tier universities performed multi-stage annotation. Annotators solved items, flagged correctness, and assigned fine-grained error types with written justifications. Dual independent review and expert adjudication were used for disagreements, ensuring high-quality labels.
Dataset Statistics: The benchmark comprises 2,000 correct questions, 2,000 erroneous questions (balanced across the four error types), and 2,000 annotated answers (610 erroneous). Inter-annotator agreement (Fleiss' $\kappa$ ) ranges from 0.69 to 0.91.

Evaluation Framework

The benchmark evaluates four sub-tasks:

Q-Detect: Binary detection of question errors.
A-Detect: Binary detection of answer errors.
Q-Type: 4-way classification of question error types.
A-Type: 3-way classification of answer error types.

Experiments were conducted on 14 state-of-the-art LLMs (6 closed-source, 8 open-source ranging from 1.5B to 72B parameters) and 3 Process Reward Models (PRMs).

Key Results

1. Performance Gap and Saturation

Even advanced systems, including GPT-o3 and DeepSeek-R1, remain far from saturating the benchmark.

Question Detection: Closed-source models like Claude-3.5-Sonnet and GPT-o1 achieved the strongest results.
Error-Type Classification: Performance is significantly lower than detection. On the SIMPLE split, the best Macro-F1 for question type classification was ~77.6% (Qwen2.5-72B), while answer type classification dropped to ~55% for the best models.
Scaling: Fine-grained type understanding is capacity-sensitive. For Qwen2.5, Macro-F1 on question type detection scaled monotonically from 35.7% (1.5B) to 77.6% (72B) on SIMPLE questions.

2. The Solving-vs-Verifying Gap

A consistent observation is that math-tuned and long-chain-reasoning models often underperform their general-purpose siblings on verification tasks.

Reasoning Models Lag: DeepSeek-R1 and Qwen2.5-Math-72B trailed their general-purpose counterparts (e.g., Qwen2.5-72B-Instruct) in error detection and typing, despite their strong performance on standard reasoning benchmarks.
Hypothesis: The authors suggest that training optimized for producing long, complex solutions may not directly optimize for concise verification, leading to a "solving-vs-verifying gap."

3. Error-Type-Aware Correction

The paper demonstrates that fine-grained error-type annotations provide actionable supervision for correcting erroneous data.

When models (DeepSeek-R1 and GPT-4o) were provided with ground-truth error types during correction tasks, accuracy improved significantly.
The largest gains were observed for structurally ambiguous errors: "Lack of conditions" saw a +4.7 percentage point gain for GPT-4o, and "Contradictions" saw a +2.1 gain. These improvements were statistically significant ( $p < 0.01$ ) via paired bootstrap testing.

4. Failure Analysis

Case studies reveal that strong LLMs often follow locally valid reasoning steps but miss global consistency checks. For instance, in a geometric problem, a model (GPT-o1) accepted a condition as a plausible alternative without verifying that the other constraints uniquely determined a contradictory angle, failing to detect the global inconsistency.

Significance and Claims

The paper positions MathDebugger as a critical tool for the data-centric AI community:

First of its Kind: It is the first text-only, type-aware benchmark covering both question-side and answer-side errors with a fine-grained taxonomy.
Diagnostic Value: The benchmark highlights that current SOTA systems are far from mastering the verification of synthetic math data, particularly in identifying specific error types.
Practical Utility: Beyond evaluation, the work demonstrates that error-type annotations are not merely diagnostic labels but serve as effective supervision signals for correcting synthetic data, directly addressing the challenge of data quality in LLM training.
Open Resource: The authors provide the dataset, source code, and project website to facilitate further research into data auditing and synthesis.

The authors conclude that while synthetic data scaling is essential, rigorous auditing via benchmarks like MathDebugger is necessary to ensure the reliability of the training corpus, and that current models still require significant improvement in verification capabilities.