🤖 AI

MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries

The paper introduces MathlibPR, a benchmark derived from real Lean/Mathlib4 pull request histories, to evaluate the ability of LLMs and agents to distinguish merge-ready contributions from non-merged ones, revealing their current struggles and highlighting the benchmark's potential for developing reviewer assistants and reward models.

Original authors: Zixuan Xie, Xinyu Liu, Shangtong Zhang

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zixuan Xie, Xinyu Liu, Shangtong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, living library of mathematics called Mathlib. It's not just a book; it's a giant, shared construction site where mathematicians and computer scientists build a perfect, error-free foundation for all of math. To keep this library safe and useful, every new piece of code (a "Pull Request" or PR) must pass two tests:

The "Does it work?" test: Does the code actually run without crashing? (The computer checks this).
The "Is it a good citizen?" test: Does the code fit in with the rest of the library? Is it written in the right style? Is it clear enough for others to use? (Humans check this).

For a long time, Artificial Intelligence (AI) has been great at passing the first test. It can write code that runs perfectly. But the second test—the human review—has become a bottleneck. There are too many submissions, and not enough human reviewers to check if the code is truly ready to be merged into the library.

This paper asks a simple question: Can AI learn to be the reviewer? Can an AI look at a piece of code that already works and decide if it's "merge-ready" or if it needs more work?

To find out, the authors created a new test called MATHLIBPR.

The Experiment: A "Blind Taste Test" for Code

Think of MATHLIBPR as a blind taste test for a new recipe.

The Setup: The researchers took real history from the Mathlib library. They gathered thousands of code submissions that had already passed the "Does it work?" test (they compiled successfully).
The Challenge: They gave these code snippets to various AI models (like DeepSeek, Qwen, and others) and asked: "Is this ready to be published in the library, or should it be sent back for revisions?"
The Catch: The AI didn't know the final outcome. It couldn't ask the human reviewers, "Did you like this?" It had to judge solely based on the code itself, just like a human reviewer would.

They tested the AI in three rounds, giving it more and more clues:

Round 1: Just the code changes and a few style guides.
Round 2: The code plus a list of automated "linting" errors (like a spell-checker for code).
Round 3: The code, the errors, plus the author's description of what they were trying to do.

The Results: The AI Got Stuck

The results were surprising and a bit disappointing for the AI community.

The AI couldn't tell the difference. Even with all the extra clues, the AI models struggled to distinguish between code that was eventually accepted and code that was rejected or sent back for fixes.
The "Yes" Bias: Most of the AIs were too optimistic. They tended to say, "Yes, this is great!" even when the code was actually messy or didn't fit the library's style. They rarely said, "No, this needs work."
The "I Don't Know" Option: Some models, when faced with a tough decision, just said, "I'm not sure." While honest, this doesn't help the library move forward.
More Context Didn't Help Much: Giving the AI more information (like the author's intent or automated error reports) didn't significantly improve its ability to make the right call.

One interesting finding was that even when the AI looked at the same project at two different times (once when it was messy, and once when it was fixed and accepted), it often couldn't tell which version was the "better" one. It was like a student taking a test on a topic they studied, but failing to notice the difference between a rough draft and the final essay.

Why This Matters

The paper concludes that while AI is excellent at writing code that works, it is currently very bad at reviewing code to see if it belongs in a high-quality library.

The authors aren't saying AI should replace human reviewers. Instead, they see this benchmark (MATHLIBPR) as a starting point. It's a tool to help train future AI systems to be better "assistant reviewers." The goal is to build an AI that can help humans by spotting obvious style issues or missing documentation, acting as a first line of defense so human reviewers can focus on the hardest, most creative parts of the job.

In short: AI is a great builder, but right now, it's a terrible inspector. This paper provides the first real test to measure exactly how bad it is, so we can teach it to do better.

Technical Summary: MATHLIBPR – A Benchmark for Formal Mathematical Library Review Readiness

1. Problem Definition

The ecosystem of Lean and Mathlib has become the standard for large language model (LLM) assisted formal reasoning, with recent successes in Olympiad-level problem solving (MiniF2F), competition mathematics (AlphaProof), and large-scale formalization projects (Gauss agent). However, these systems primarily consume Mathlib as a dependency rather than contributing to it. While LLM-generated code can be verified for correctness by the Lean kernel, determining whether a contribution is "merge-ready"—i.e., adhering to community conventions, maintaining library coherence, and being suitable for long-term maintenance—requires human review.

Human review is currently the primary bottleneck for Mathlib's growth, with a backlog of approximately 300 pull requests (PRs) and median wait times of two weeks. The central question addressed by this paper is: Can LLMs help review Mathlib PRs? Specifically, can they distinguish between build-passing code that is ready for integration and build-passing code that requires revision or will be rejected?

2. Methodology

2.1 Dataset Construction: MATHLIBPR

The authors introduce MATHLIBPR, a benchmark constructed from real Mathlib4 PR histories.

Source: Closed PRs from the leanprover-community/mathlib4 repository (as of April 5, 2026).
Selection Criteria: The benchmark focuses exclusively on build-passing snapshots. This eliminates trivial strategies based on syntax errors or compilation failures, forcing models to judge based on code quality and conventions.
Labels:
- Merge Ready: Snapshots accepted into the master branch.
- Not Merge Ready: Snapshots that passed build checks but were either revised before acceptance or belonged to PRs that were never merged.
Composition: The final dataset contains 15,895 labeled snapshots from 12,063 PRs (11,409 merge-ready, 4,486 not-merge-ready). It includes 3,687 within-PR pairs (an earlier build-passing snapshot vs. the final accepted version) to test recognition of revision progress.

2.2 Evaluation Protocol

The authors propose a three-stage evaluation protocol that progressively supplies richer context to the models:

Stage 1: Code-local evidence (diff, changed files) and compact digests of Mathlib guidelines (naming, style, documentation).
Stage 2: Adds structured diagnostics (linting, imports, declaration placement, API fit) derived from historical GitHub check records or locally backfilled.
Stage 3: Adds PR title and description as stated intent.

2.3 Systems Evaluated

The benchmark evaluates two classes of systems:

LLM Models: DeepSeek-R1-Distill-Qwen, Qwen3-8B, Goedel-Prover-V2-32B (pre-fine-tuned on Mathlib), and Kimina-Prover-Distill-8B.
LLM Agents: Codex (GPT-5.4) and Claude Code (Claude Sonnet 4.6). Agents operate in a read-only local repository checkout, restricted from editing files, accessing the web, or retrieving PR outcomes.

2.4 Output Schema and Metrics

Systems are required to output a structured JSON review rather than free-form text. The schema includes:

Verdict: merge_ready, not_merge_ready, or uncertain.
Score: A scalar merge-readiness score ( $p_{merge\_ready}$ ).
Axes: Assessments across eight rubric dimensions (naming, documentation, local structure, file placement, imports, proof readability, API fit, repository overlap).

Metrics:

MR/NMR Recall: Recall for merge-ready and not-merge-ready classes (treating uncertain and invalid outputs as errors).
Balanced Accuracy: Mean of MR and NMR recall.
Valid-Output Rate: Percentage of schema-valid outputs within runtime budgets.
AUROC: Area under the ROC curve based on scalar scores.

3. Key Results

3.1 Overall Performance

Low Accuracy: No system reliably distinguishes merge-ready from not-merge-ready snapshots. The best-performing model (DeepSeek) achieved a Balanced Accuracy of 36.0% at Stage 3, significantly below the 50% chance baseline for a balanced task (though the dataset is imbalanced, the recall metrics highlight the failure).
Class Asymmetry: Systems exhibit a strong bias toward predicting merge_ready.
- DeepSeek (Stage 3): 69.8% MR Recall, but only 2.3% NMR Recall.
- Qwen (Stage 3): 48.3% MR Recall, 0.0% NMR Recall.
- Codex (Agent): 92.0% MR Recall, but only 12.0% NMR Recall.
Impact of Context: Adding diagnostics and PR intent (Stage 2 and 3) yielded limited and uneven gains. For instance, DeepSeek's balanced accuracy improved from 18.1% to 36.0%, but its ability to identify not-merge-ready code remained near zero.

3.2 Agent vs. Model Performance

Agents: While Codex showed high operational reliability (99.9% valid output rate) and high MR Recall, it failed to identify non-merge-ready code. Claude Code had lower reliability (56.4% valid rate) and similar poor NMR recall.
Specialized Models: Goedel and Kimina, despite being trained on formal proofs, performed poorly. Goedel's valid output rate collapsed from 68.3% (Stage 1) to 13.5% (Stage 3), and both models frequently produced invalid outputs (prose or code blocks) instead of the required JSON schema.

3.3 Within-PR Analysis

In a controlled comparison of earlier build-passing snapshots vs. final accepted snapshots from the same PR:

Models barely beat chance. DeepSeek's pairwise accuracy plateaued at 51.1% (Stage 3), and Qwen remained near 49.1%.
This indicates that models cannot reliably detect the specific improvements that led to a PR's acceptance, even when comparing two versions of the same contribution.

3.4 Error Analysis

Abstention: Systems often abstain (uncertain) on documentation or proof readability axes.
Invalid Outputs: A significant portion of outputs from specialized models (Goedel, Kimina) and agents (Claude Code) were invalid due to schema violations, timeouts, or failure to produce JSON. This suggests a mismatch between the models' training objectives (theorem proving) and the structured review interface.

4. Contributions

MATHLIBPR Benchmark: The first benchmark for evaluating LLMs on the merge-readiness of build-passing Mathlib contributions, derived from real PR histories.
Staged Evaluation Protocol: A protocol that systematically tests whether richer context (diagnostics, PR intent) improves reviewer-like judgment.
Empirical Study: A comprehensive evaluation showing that current LLMs and agents struggle to distinguish merge-ready code from build-passing code that requires revision, highlighting a gap between code correctness and library integration quality.

5. Significance and Claims

The paper claims that MATHLIBPR provides a step toward reviewer assistants and reward models for formal mathematical libraries.

Current State: The results demonstrate that judging merge-readiness is a substantially harder capability than detecting syntax errors or compilation failures. Current systems fail to capture the nuanced review norms required to curate an integrated mathematical library.
Future Utility: The benchmark is intended to be used for:
- Reviewer Assistance: Triage and surfacing likely blockers for human reviewers.
- Reward Modeling: Training reward models using the within-PR pair structure (preferring the final accepted snapshot over an earlier build-passing one) to steer LLMs toward generating code that is not only correct but also maintainable and merge-ready.
Limitations: The authors acknowledge that the benchmark covers a selective slice of PR history and that labels are derived from historical maintainer decisions, which may include non-technical factors. They explicitly state the goal is not to replace human reviewers but to provide an advisory signal.

In conclusion, while LLMs have succeeded in generating correct proofs, the transition to contributing to Mathlib requires a level of judgment regarding code style, documentation, and library fit that current models have not yet mastered. MATHLIBPR establishes the baseline for measuring progress in this direction.