Technical Summary: U-MATH and µ-MATH Benchmarks

Problem Statement

Current evaluations of mathematical reasoning in Large Language Models (LLMs) are constrained by significant limitations in existing benchmarks. Most prominent datasets, such as GSM8K and MATH, focus primarily on school-level problems, leading to rapid saturation and failing to assess advanced reasoning capabilities. Furthermore, existing benchmarks often rely on multiple-choice formats, which simplify tasks and allow models to exploit hints, or they are limited in scope to niche Olympiad-style problems, neglecting the practical middle-ground of university-level coursework.

Additionally, the assessment of multi-modal mathematical reasoning (incorporating visual elements like graphs and diagrams) lacks robust, open-ended datasets. Finally, the evaluation of free-form mathematical solutions is inherently difficult due to expression ambiguity. While LLMs are increasingly used as "judges" to evaluate these solutions, their own reliability is often unquantified, and existing meta-evaluation datasets are largely based on the same limited school-level benchmarks they aim to critique.

Methodology

To address these gaps, the authors introduce two interconnected benchmarks: U-MATH and µ-MATH.

1. U-MATH (University Math)

U-MATH is a benchmark comprising 1,100 unpublished, open-ended problems sourced from current US university curricula.

• Curation: Problems were collected from Gradarius, a platform providing content for top US universities. The selection process involved filtering out calculator-dependent problems and short solutions to prioritize reasoning over arithmetic. A diverse set of smaller models was used to identify the most challenging problems, which were then manually curated by subject matter experts and academic faculty from the Stevens Institute of Technology.
• Composition: The dataset covers six core subjects: Algebra, Differential Calculus, Integral Calculus, Multivariable Calculus, Precalculus, and Sequences and Series. Approximately 20% of the problems include visual elements (graphs, tables, geometric figures), while the remaining 80% are text-only.
• Format: All problems are free-form, requiring multi-step solutions rather than multiple-choice selection.

2. µ-MATH (Meta U-MATH)

µ-MATH is a meta-evaluation benchmark designed to assess the quality of LLM judges.

• Construction: It consists of 1,084 samples derived from a subset of U-MATH problems (approximately 25%).
• Data Generation: For each problem, four solutions were generated by different top-performing models (Qwen2.5 72B, Llama-3.1 8B, GPT-4o, and Gemini 1.5 Pro).
• Ground Truth: Solution correctness was determined via a combination of manual labeling by experts and automatic verification using the Gradarius-API to test formal equivalence of mathematical expressions.
• Task: The benchmark tasks LLMs with evaluating a generated solution against a reference answer and a problem statement, outputting a binary "Yes" or "No" verdict.

Key Contributions

1. U-MATH Dataset: The release of 1,100 university-level, open-ended mathematical problems, with 20% incorporating visual elements, addressing the lack of scope in current benchmarks.
2. µ-MATH Dataset: A meta-evaluation framework enabling the rigorous assessment of LLM judges on university-level tasks, moving beyond the limitations of GSM8K/MATH-based meta-datasets.
3. Comprehensive Analysis: A comparative evaluation of leading open-source and proprietary LLMs on both problem-solving and judgment tasks, revealing distinct behavioral patterns and performance gaps.

Experimental Results

Problem-Solving Performance (U-MATH)

• Reasoning Models: Reasoning-focused models (e.g., o1, o3-mini, DeepSeek-R1) significantly outperform standard inference models. The best-performing model, o1, achieved 93.1% accuracy on text-only tasks and 58.5% on visual tasks.
• Text vs. Visual Gap: There is a marked limitation in multi-modal reasoning. While top models reach ~93% on text, performance drops to ~58.5% on visual tasks. Furthermore, transitioning to visual inputs often degrades textual performance in non-reasoning models.
• Open vs. Proprietary: Open-weight models are catching up on text-only tasks (e.g., DeepSeek-V3 at 69.3% vs. Gemini 1.5 Pro at 71.7%) but lag significantly in visual tasks, where proprietary models (specifically the Gemini family) dominate.
• Specialization vs. Size: Specialized models (e.g., Qwen2.5-Math 7B) can outperform much larger generalist models (e.g., Llama-3.1 70B), and continuous fine-tuning (e.g., Llama-3.1 Nemotron) yields measurable improvements.

Judgment Performance (µ-MATH)

• Inherent Difficulty: Evaluating mathematical solutions is non-trivial. Even the best-performing models achieve a maximum F1-score of 90.1%, indicating that current LLM judges remain fallible even with ground truth access.
• Dissociation of Skills: Superior problem-solving ability does not guarantee superior judgment. A trade-off exists between reasoning coherence and judgment performance.
  • Conservative Judges: Models like Claude 3.5 Sonnet exhibit high True Negative Rates (TNR) but lower True Positive Rates (TPR), often rejecting correct but complex derivations.
  • Lenient Judges: Models like Qwen-Math exhibit high TPR but lower TNR, often accepting incorrect solutions due to hallucination or lack of structural adherence.
• Prompting Effects: Switching from Automatic Chain-of-Thought (AutoCoT) to manual Chain-of-Thought (CoT) significantly improves judgment performance for non-reasoning models (e.g., >10% F1 gain for Gemini 1.5), though reasoning models remain largely unaffected by prompt changes.
• Bias: Judges exhibit model-specific biases, performing better on solutions generated by models similar to themselves (e.g., Llama judges favoring Llama solutions) and worse on others.

Significance and Claims

The paper posits that U-MATH and µ-MATH are essential for advancing the evaluation of LLMs in complex, real-world mathematical domains. The authors claim that:

1. Current Benchmarks are Insufficient: Existing datasets fail to capture the depth of university-level reasoning and the challenges of multi-modal integration.
2. Judgment is a Distinct Skill: The ability to solve a problem and the ability to judge a solution are distinct capabilities, often exhibiting a trade-off in model behavior.
3. Meta-Evaluation is Critical: Quantifying the errors and biases of LLM judges is necessary to ensure the reliability of automated evaluations. The authors demonstrate that without rigorous meta-evaluation, evaluation results may be skewed by the judge's inherent biases.
4. Future Directions: The authors suggest that while reasoning models show promise, further research is needed to address the saturation of current metrics, improve visual reasoning, and develop assessment protocols that allow for partial credit and finer-grained analysis.

The authors emphasize that their work is a first step in quantifying auto-evaluation errors and that the open-sourcing of these datasets and code is intended to facilitate reproducibility and further research into robust mathematical reasoning and evaluation.