Technical Summary: AdvancedMathBench

1. Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable proficiency in high-school and competition-level mathematics, their capabilities in advanced mathematics (undergraduate and doctoral levels) remain poorly understood. Existing benchmarks suffer from two primary limitations:

1. Limited Scope: Most benchmarks focus on competition-style problems or calculation-heavy tasks, lacking coverage of undergraduate courses, doctoral qualifying exams, and research-level reasoning.
2. Insufficient Evaluation Granularity: Current evaluations rely heavily on final-answer correctness or coarse solution matching. This approach fails to assess the validity of the underlying reasoning process. A model may produce a correct final answer via flawed logic, or a valid proof may deviate from a reference solution, making template matching unreliable.

Consequently, there is a need for a rigorous framework that evaluates not just whether a model can solve a problem, but whether it can construct and verify complete, natural-language mathematical proofs with rigorous intermediate claims.

2. Methodology: AdvancedMathBench

The authors introduce AdvancedMathBench, a benchmark suite designed to evaluate LLMs on advanced mathematical proof generation and verification. The suite comprises two complementary components: ProverBench and VerifierBench, supported by a dedicated automatic verification pipeline.

2.1 Data Curation

The dataset construction follows a multi-stage pipeline (Figure 2):

• Sourcing: Problems are collected from four sources: undergraduate exams from leading Chinese universities, doctoral qualifying exams (QE) from top global institutions, the S.-T. Yau College Student Mathematics Contest, and textbook exercises.
• Filtering: Answer-centric formats (multiple-choice, fill-in-the-blank) are removed. Only problems requiring rigorous proof or complete reasoning processes are retained.
• Difficulty Pre-screening: Using verifier uncertainty (entropy of repeated verification outcomes by top-tier LLMs), the authors filter out trivial problems and retain "plausible but error-prone" proof trajectories that induce uncertainty.
• Expert Quality Control: PhD-level mathematicians review problem statements, reference solutions, and candidate proofs to ensure clarity, correctness, and the removal of ambiguous or ill-posed samples.

2.2 ProverBench (Proof Generation)

• Composition: Contains 245 problems divided into two splits:
  • UG (Undergraduate): 200 problems covering core subjects (probability, analysis, algebra, etc.).
  • QE (Doctoral Qualifying Exam): 45 problems covering advanced topics (graph theory, topology, applied math).
• Task: Models must generate step-by-step natural-language proofs.
• Evaluation: Since proofs cannot be validated by final answers alone, an automatic verification pipeline (trained on large-scale expert annotations) evaluates the full proof trajectory for correctness, error localization, and fine-grained quality.

2.3 VerifierBench (Proof Verification)

• Composition: Contains 888 problem-proof-ground-truth triplets. Each proof is annotated by experts who judge validity, provide detailed analysis, and localize errors.
• Annotation Protocol: Unlike "first-error-only" labeling, experts analyze the entire proof chain, distinguishing between Fatal Errors (invalidating the global argument) and Recoverable Errors (local slips).
• Task: Models must judge proof validity, provide a verification rationale, and identify the index of the first error.
• Evaluation: A Meta-Verifier (trained on expert ground truth) evaluates model outputs based on agreement with experts regarding validity judgments, error coverage, and rationale quality.

2.4 Automatic Verification Pipeline

To evaluate ProverBench at scale, the authors developed a specialized verifier trained via:

1. Large-scale Annotation: Generating diverse proof trajectories and selecting informative cases via verifier uncertainty.
2. Positive-sample Augmentation: Repairing erroneous proofs using expert suggestions to balance the training distribution and prevent the verifier from becoming overly pessimistic.
3. RL with Meta-verification Reward: Training the verifier using Reinforcement Learning (GRPO) where the reward is determined by a Meta-Verifier's alignment with expert rationales, not just binary correctness.
4. Pessimistic Verification: A proof is accepted only if all parallel verification passes judge it correct, ensuring high precision in accepting valid proofs.

3. Key Results

3.1 Proof Generation (ProverBench)

• Performance Gap: Frontier models struggle significantly with advanced proofs. The best-performing model, GPT-5.5-xhigh, achieves only 64.5% on the UG split and 48.9% on the harder QE split.
• Difficulty Gradient: There is a substantial performance drop from UG to QE levels across all models. For instance, Gemini-3.1-Pro-Preview drops from 46.5% (UG) to 17.8% (QE).
• Comparison: Scores on ProverBench are substantially lower than on answer-centric benchmarks (e.g., HMMT) or competition-style proof benchmarks (e.g., USAMO), indicating that existing benchmarks fail to expose the limitations of models in advanced reasoning.

3.2 Proof Verification (VerifierBench)

• Verification Bottleneck: The best model, DeepSeek-V4-Pro, achieves a Balanced F1 of 65.1 under Meta-Verification.
• Over-estimation by Rough Evaluation: Simple binary validity checks (Rough scores) significantly overestimate performance. Moving to Meta-Verification (which evaluates rationales) lowers the average Balanced F1 by 9.0 points.
• Error Detection Failure: Models exhibit a strong bias toward over-accepting plausible but invalid proofs. Many models achieve high True Positive Rates (TPR > 90%) but very low True Negative Rates (TNR < 40%), indicating they fail to detect subtle logical gaps.
• Verifier Performance: The authors' automatic verifier outperforms general LLM-as-Judge baselines, achieving a 73.9 Balanced F1 on held-out examples compared to ~61-63% for frontier models acting as judges.

4. Key Contributions

1. AdvancedMathBench Suite: A comprehensive benchmark covering undergraduate and doctoral-level mathematics with a focus on natural-language proof generation and verification, filling a gap left by competition-style and answer-centric benchmarks.
2. Expert-Aligned Verification Pipeline: A scalable, automatic verification system trained on large-scale expert annotations that provides fine-grained error localization and rationale assessment, outperforming standard LLM judges.
3. Dual-Task Evaluation: The introduction of VerifierBench to explicitly test a model's ability to critique and verify proofs, revealing that current models are significantly better at generating plausible arguments than at detecting subtle errors within them.
4. Methodological Rigor: The use of pessimistic verification and rationale-aware rewards to ensure that evaluation metrics reflect true mathematical validity rather than superficial pattern matching.

5. Significance and Claims

The paper claims that AdvancedMathBench provides a rigorous foundation for diagnosing the capabilities of LLMs in advanced mathematical reasoning. The results indicate that:

• Frontier models are far from saturating advanced mathematical reasoning; substantial room for improvement exists, particularly at the doctoral level.
• Proof verification remains a major bottleneck. Models often fail to reject invalid proofs that appear plausible, suggesting that current "verifiers" are not yet reliable enough for autonomous mathematical discovery without human oversight.
• Evaluation methodology matters: Relying on final answers or binary validity judgments leads to an inaccurate estimation of model capabilities. Process-level assessment, including rationale quality and error localization, is essential for a true understanding of mathematical reasoning.

The authors conclude that future work must focus on developing verifier-aware proof generation and proof verifiers capable of reliably identifying subtle mathematical failures, rather than merely producing plausible conclusions.