Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs

1. Problem Statement

The paper addresses the critical limitation of current Large Language Model (LLM) evaluation in mathematics: the reliance on static benchmarks. Traditional benchmarks (e.g., MATH, GSM8K) suffer from three main issues:

• Rapid Saturation: Models quickly master fixed datasets, rendering them ineffective for distinguishing frontier capabilities.
• Narrow Scope: They often focus solely on "final-answer" problems, failing to capture complex mathematical reasoning, proof generation, or research-level tasks.
• Lack of Maintenance: Once published, benchmarks are rarely updated, leading to stale evaluations that do not reflect the rapid pace of model improvement.

The authors argue that the field requires a shift from one-time benchmark releases to continuously maintained evaluation platforms that can adapt to model capabilities, cover diverse mathematical domains, and provide transparent, up-to-date analysis.

2. Methodology: MATHARENA as an Evaluation Platform

The authors introduce MATHARENA, an evolution of their previous benchmark into a comprehensive evaluation platform. The methodology is built on three core design principles distinguishing it from static benchmarks:

A. Dynamic Lifecycle (Changing Tasks)

• Continuous Updates: Benchmarks are retired when saturated and new ones are introduced as models improve.
• Diverse Task Categories: The platform covers four distinct capability dimensions:
  1. Final-Answer Competitions: Scalable, automatically verifiable problems (e.g., AIME 2026, HMMT Feb 2026, Kangaroo 2025 for visual reasoning, and the curated "Apex" set of ultra-difficult problems).
  2. Proof-Based Competitions: Human-graded or semi-automated evaluations of formal proofs (e.g., USAMO 2026, Putnam 2025, Miklós Schweitzer 2025, IMC 2025).
  3. Research-Level Benchmarks: Problems extracted from recent arXiv papers, including:
     • ArXivMath: Final-answer questions derived from paper abstracts.
     • BrokenArXiv: A reliability test where models must identify and refuse to prove false statements derived from real papers.
     • ArXivLean: Formal proof generation in the Lean theorem prover for research-level theorems.
  4. Project Euler: Weekly updates of programming/math problems.

B. Robust Evaluation Protocols

• Human-in-the-Loop: For proof-based tasks, the platform utilizes official competition organizers (e.g., Putnam, Miklós Schweitzer) or expert human graders to ensure high-quality evaluation.
• LLM Juries: For semi-automatic grading (e.g., USAMO), the platform employs a jury of multiple frontier LLMs (GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6) to reduce bias, with final manual verification.
• Faithful Measurement: The platform rigorously controls for implementation details (e.g., token limits, tool usage, prompt formatting) to ensure scores reflect true capability rather than engineering artifacts.
• Expected Performance Metric: To compare models across different subsets of benchmarks without running every model on every task (which is cost-prohibitive), the platform uses Item Response Theory (IRT) to impute missing scores and calculate an aggregate "expected performance."

C. Transparency and Public Interface

• All results, model outputs, metadata, and code are publicly available at matharena.ai.
• The interface allows users to inspect individual problem traces, compare models across specific categories, and track cost metrics.

3. Key Contributions

1. Paradigm Shift: The paper formally argues for and demonstrates the utility of evaluation platforms over static benchmarks, defining the necessary design principles (changing tasks, ongoing processes, public interfaces).
2. Platform Expansion: MATHARENA is substantially extended beyond its original scope to include proof-based competitions, research-level arXiv problems, and formal Lean proofs.
3. New Benchmarks: Introduction of novel benchmarks:
   • Apex/Apex Shortlist: Curated, unsolved problems from 2025 competitions.
   • BrokenArXiv: A specific test for "sycophancy" (the tendency of models to agree with false premises).
   • ArXivLean: A benchmark for formalizing and proving research-level math in Lean.
4. Comprehensive Analysis: A detailed performance analysis of frontier models (GPT-5.5, DeepSeek-v4-Pro, etc.) across these diverse domains.

4. Key Results

• Rapid Progress: Average performance across all benchmarks has risen from 45% (May 2025) to 84% (current), driven primarily by models like GPT-5.5.
• Saturation of Easy Tasks: Frontier models (e.g., GPT-5.5) achieve near-perfect scores (97%+) on standard final-answer competitions (AIME, HMMT), indicating these are no longer useful for differentiating top-tier models.
• Proof and Research Capabilities:
  • GPT-5.5 leads all models, achieving 98.2% on USAMO 2026 and 74.1% on ArXivMath.
  • Open Models: DeepSeek-v4-Pro is the strongest open model but lags behind GPT-5.5 by ~20% on average, with a larger gap on non-final-answer tasks (proofs and research).
• Reliability Failures (BrokenArXiv):
  • Models frequently fail to recognize false statements. GPT-5.5 achieves 71.7% (the only model with strong performance), while most others score below 20%. This highlights a critical safety gap: models often "sycophantically" prove false claims.
• Formal Proof Challenges (ArXivLean):
  • Performance remains low (17.1% for GPT-5.5), indicating that while models can solve research problems, they struggle to formalize them in Lean without significant scaffolding or cost.
• Visual Reasoning: Models struggle more with visual math (Kangaroo 2025) than textual final-answer problems, often failing to reason directly in image space.

5. Significance and Future Implications

• Tracking Frontier Progress: MATHARENA provides the first continuously updated, transparent view of LLM capabilities in high-level mathematics, moving beyond the "snapshot" nature of academic papers.
• Safety and Reliability: The BrokenArXiv results serve as a stark warning for the deployment of LLMs in research; without rigorous verification, models may confidently generate incorrect proofs, potentially polluting the scientific literature.
• Benchmarking Best Practices: The paper establishes a gold standard for evaluation, emphasizing the need for human verification, diverse task types (proofs, formalization, reliability), and the mitigation of evaluation biases (e.g., via LLM juries and manual review).
• Adoption: The platform has already been adopted by major model developers (OpenAI, Google, Anthropic, DeepSeek) for their internal model reports, validating its role as a shared, up-to-date standard for the community.

In conclusion, MATHARENA demonstrates that while LLMs are rapidly mastering standard mathematical problems, significant challenges remain in proof quality, research-level reliability, and formal verification. The platform serves as a critical tool for monitoring these nuances and guiding future model development.