🤖 AI

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

This paper introduces PyraMathBench, a comprehensive hierarchical benchmark designed to evaluate the integration of numerical processing and mathematical reasoning in large language models, and proposes the SOLVE module and IRPO training method to significantly enhance model performance by addressing weaknesses in numerical computation and abstract reasoning.

Original authors: Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) as brilliant, well-read students who can write poetry and summarize history books perfectly. However, when you hand them a math word problem, they often stumble. They might understand the story but fail at the actual numbers, or they might "hallucinate" (make up) numbers that aren't there.

The paper "PyraMathBench" introduces a new way to test these students and a new method to help them learn. Here is the breakdown in simple terms:

1. The Problem: The "Black Box" of Math Failure

Currently, when we test AI on math, we usually just look at the final answer. Did they get it right? Yes or No.

The Flaw: If the answer is wrong, we don't know why. Did the AI misunderstand the question? Did it forget a math rule? Or did it just do the arithmetic wrong (like a calculator with a broken button)?
The Analogy: It's like grading a student's essay by only looking at the final grade. If they got a "C," you don't know if they had bad handwriting, didn't understand the prompt, or made a spelling error. You need to see the steps to fix the problem.

2. The Solution: PyraMathBench (The "Pyramid" Test)

The authors built a massive new benchmark called PyraMathBench. Think of it as a pyramid of math skills.

The Structure: Instead of just giving the AI a hard problem, they break it down. They take 7,404 real-world math stories and slice them into 32,505 smaller pieces (subtasks).
The Layers:
- The Base (Numerical Parsing): Can the AI find the numbers in the text? Can it read numbers off a picture?
- The Middle (Understanding & Calculation): Can it turn the story into a math equation? Can it actually do the addition or solve the equation?
- The Top (Complex Reasoning): Can it put it all together to solve the original hard problem?
The Result: By testing the AI on every single layer, the researchers found that many top-tier AIs are actually terrible at the basics. They often fail to recognize numbers in images or get confused by simple arithmetic, even if they are smart at logic.

3. The Diagnosis: What's Wrong?

After testing 11 different AI models (including big names like GPT-4, Llama, and DeepSeek), they found two main weaknesses:

Bad at "Reading" Numbers: The models often miss numbers in text or, worse, in images. They might see a clock in a picture but guess the time wrong, or they might ignore a crucial number in a word problem.
Hallucinating: When they don't know a number, they sometimes just make one up to keep the story going.

4. The Fix: SOLVE and IRPO

To help the AI get better, the authors created two new tools, like a smart tutor and a personal trainer.

SOLVE (The Smart Tutor):
- The Problem: AI often tries to use external tools (like a calculator or a code interpreter) but fails because it formats the request poorly (e.g., writing the wrong code syntax). It's like a student trying to use a calculator but pressing the buttons in the wrong order.
- The Fix: SOLVE is a module that acts as a translator. It lets the AI ask for help in any messy way it wants (even "handwritten" style), and SOLVE cleans it up and sends the perfect request to the calculator. It also knows when the AI is smart enough to solve a simple problem on its own, so it doesn't waste time calling a tool.
IRPO (The Personal Trainer):
- The Problem: Standard training methods reward the AI only for the final answer. They don't teach the AI how to use the calculator effectively during the process.
- The Fix: IRPO is a new training method. It watches the AI's entire thought process. If the AI calls a calculator at the right time, it gets a reward. If it calls a calculator for a problem it could have solved itself, it gets a penalty. It teaches the AI to know when to think and when to use a tool.

5. The Outcome

When they applied these fixes to the Qwen-2.5 model:

The model's math score jumped by 5.0 points.
It became much better at knowing when to use a calculator and when to do the math itself.

Summary

The paper argues that to make AI better at math, we can't just look at the final score. We need to break math down into a pyramid of tiny skills to see exactly where the AI is failing. Once we see the cracks (like bad number recognition), we can build smart tools (SOLVE) and better training (IRPO) to fix them, turning a confused student into a math whiz.

Technical Summary: PyraMathBench

Problem Statement

Despite the pivotal role of numerical reasoning in applications ranging from scientific research to financial analysis, current Large Language Models (LLMs) struggle significantly with mathematical tasks. Existing benchmarks often fail to disentangle numerical processing flaws (e.g., tokenization errors, calculation mistakes, hallucinations) from mathematical reasoning errors (e.g., logical misinterpretation). Consequently, when a model fails, it is unclear whether the error stems from an inability to compute or an inability to understand the problem structure. Furthermore, existing frameworks often test abilities in isolation, lacking cross-task correlations that reveal how foundational skills influence complex reasoning. Additionally, while Multimodal LLMs (MLLMs) are evaluated on visual math problems, current assessments often overestimate their visual reasoning capabilities by not distinguishing between reliance on textual cues versus actual visual data extraction.

Methodology

1. PyraMathBench (PMB)

The authors introduce PyraMathBench, a comprehensive hierarchical benchmark designed to evaluate the synergy between numerical processing and mathematical reasoning.

Scale and Structure: The benchmark comprises 32,505 questions derived from 7,404 math word problems (MWPs) and visual reasoning problems.
Hierarchical Taxonomy: Inspired by Piaget's cognitive theory, tasks are decomposed into 4 key cognitive aspects and 14 distinct subtasks:
1. Complex Reasoning: Math Word Problems (MWP) and Visual Reasoning Problems (VRP).
2. Understanding: Question Abstraction (QA), Task Decomposition (TD), and Math Knowledge (MK).
3. Calculation: Arithmetic, Equation Solving (ES), Sorting, and Formula Application (FA).
4. Numerical Parsing: Number Conversion (NC), Unit Conversion (UC), Numeral Recognition (NR), Visual Data Quantification (VDQ), and Object Counting (OC).
Data Construction: Data is sourced from six existing datasets (e.g., GSM8K, MATH, MathVista) and practice collections. Subtasks are annotated by six experts proficient in high school mathematics, ensuring rigorous decomposition of real-world problems rather than synthetic generation. The dataset includes both unimodal and multimodal tasks.
Evaluation Metrics: A custom scoring function based on relative error (0–100 scale) is used for numerical answers, while MathBERT embeddings and cosine similarity are used for text answers.

2. Proposed Solutions: SOLVE and IRPO

To address the identified weaknesses in numerical computation and tool utilization, the authors propose two enhancements:

SOLVE (Smart Optimization & Learning-based VErsatile module): A tool integration module designed to improve flexibility and adaptability.
- Fuzzy Call Recognition: Unlike rigid "plug-and-play" modules, SOLVE accepts tool calls in various formats (LaTeX, Python, informal text) and standardizes them, reducing parsing failures.
- Dynamic Difficulty Assessment: Utilizing Item Response Theory (IRT-2PL), SOLVE estimates the difficulty of a task and the model's current proficiency ( $\theta$ ). If the probability of the model solving the task correctly exceeds a threshold (0.95), SOLVE bypasses the tool call to optimize efficiency.
IRPO (Interactive Relative Policy Optimization): A new Reinforcement Learning (RL) framework extending Group Relative Policy Optimization (GRPO).
- Dual-Stream Sampling: IRPO simultaneously samples outputs from a base policy ( $\pi_\theta$ ) and a tool-augmented policy ( $\pi^\tau_\theta$ ).
- Reward Function: The reward function is multi-faceted, penalizing unnecessary tool calls when the model is capable (based on subtask accuracy) and rewarding tool usage when it aids in solving complex sub-problems. It incorporates a dynamic baseline and a logistic function modeling the probability of correct tool-free solutions.

Key Results

Benchmark Evaluation

Experiments on 11 representative LLMs (including GPT-4o, DeepSeek-R1, Qwen-2.5, and Llama-3.1) reveal:

Performance Gaps: While top models like DeepSeek-R1 and GPT-4o perform well on complex reasoning (MWP), they exhibit significant weaknesses in Question Abstraction, Equation Solving, and Numerical Parsing.
Math-Specific Models: Surprisingly, math-specialized models (e.g., DeepSeek-Math, Qwen2.5-Math) underperformed compared to generalist models like Llama-3.1 on several subtasks, attributed to poor instruction-following (ignoring prompts to abstract problems) and inconsistent answer formats.
Multimodal Limitations: MLLMs scored extremely low (average < 20) on visual numerical parsing subtasks (NR, VDQ, OC). Analysis indicates they predominantly rely on textual information rather than extracting data from images, even in simple scenarios like counting objects or reading clocks.
Correlation Analysis: Pearson correlation analysis shows that performance on Question Abstraction (QA), Equation Solving (ES), and Formula Application (FA) is most strongly correlated with overall MWP performance. These areas represent the primary bottlenecks for current LLMs.

Improvement via SOLVE and IRPO

Applying the proposed methods to Qwen-2.5 and Llama-3.1-8B yielded significant improvements:

Baseline vs. PPM: Simple "Plug-and-Play" tool calling actually degraded performance (e.g., Qwen-2.5 dropped from 91.1 to 82.8) due to parsing errors and context bloat.
SOLVE Impact: Integrating SOLVE improved scores (Qwen-2.5: 95.8; Llama-3.1: 74.8) by handling fuzzy tool calls and bypassing unnecessary calls.
IRPO Impact: Further training with IRPO achieved the highest scores (Qwen-2.5: 96.1; Llama-3.1: 76.1), demonstrating a 5.0% improvement over the vanilla baseline for Qwen-2.5.
Comparison: IRPO outperformed GRPO, particularly on weaker models, by explicitly optimizing the decision-making process for tool invocation across multiple rounds.

Significance and Claims

The paper claims that PyraMathBench provides a necessary shift from "answer-centric" evaluation to process-oriented, hierarchical assessment. By decomposing complex problems into atomic cognitive tasks, the benchmark allows researchers to pinpoint specific failure modes (e.g., distinguishing a calculation error from a reasoning error).

The authors assert that their proposed SOLVE and IRPO frameworks effectively mitigate the "tool call parsing failures" and "inefficient tool usage" that currently hinder LLMs. The results suggest that enhancing the synergy between internal reasoning and external tool invocation is critical for advancing mathematical capabilities. The paper concludes that while current models show promise in high-level reasoning, their proficiency in foundational numerical processing and multimodal data extraction remains a significant barrier to solving real-world mathematical problems.

Limitations

The authors acknowledge several limitations:

Decomposition Ambiguity: Task decomposition can have multiple valid versions, potentially influencing results, particularly in the "Understanding" aspect.
Scope: The taxonomy covers 14 subtasks but does not encompass all possible atomic language capabilities.
Language: The study is currently limited to English.
Domain Specificity: The SOLVE and IRPO methods are optimized for Math Word Problems and may not generalize directly to other mathematical domains like formal proofs or pure algebraic manipulation.