💻 computer science

MathAtlas: A Benchmark for Autoformalization in the Wild

This paper introduces MathAtlas, the first large-scale autoformalization benchmark comprising approximately 52,000 graduate-level mathematical concepts from 103 textbooks, which reveals the extreme difficulty of current models in handling complex, dependency-rich research mathematics.

Original authors: Nilay Patel, Noah Arias, Davit Babayan, Victoria Cochran, Timothy Libman, Hafsah Mahmood, Liam McCarty, Soli Munoz, Laurel Willey, Jeffrey Flanigan

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nilay Patel, Noah Arias, Davit Babayan, Victoria Cochran, Timothy Libman, Hafsah Mahmood, Liam McCarty, Soli Munoz, Laurel Willey, Jeffrey Flanigan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant translator who can turn human stories into a strict, computer-readable language called "Lean." For a long time, this translator has only been tested on simple stories—like high school math problems or basic puzzles. The computer could easily translate these because the rules were simple, and the computer had already seen similar rules before.

But what happens when you ask this translator to translate a complex, graduate-level research paper? That is the problem MathAtlas tackles.

Here is a simple breakdown of what the paper does, using some everyday analogies:

1. The Problem: The "Library" is Too Big

Imagine you are trying to build a house (a formal proof) based on a blueprint (a math textbook).

Old Benchmarks: Previous tests only asked the translator to build a small shed. The materials were all right there in the toolbox, and the instructions were short.
The Real World: Graduate math is like trying to build a skyscraper. To build the top floor, you need the 50th floor. To build the 50th floor, you need the 49th, and so on, all the way down to the foundation.
The Issue: In advanced math, the "foundation" (definitions and theories) often hasn't been built in the computer's language yet. If the translator doesn't know the definition of a "Lie Algebra" (a complex concept), it can't translate the theorem that uses it.

2. The Solution: MathAtlas (The "Wild" Map)

The authors created MathAtlas, a massive new test set.

The Scale: They scraped 103 graduate-level math textbooks. It's like taking a library of 52,000 pages of advanced math and turning it into a digital map.
The "Dependency Graph": This is the paper's superpower. Imagine a "Choose Your Own Adventure" book where every page has arrows pointing to other pages you need to read first. MathAtlas draws these arrows. It shows that to understand Proposition 15, you first need to understand Dedekind Rings, which in turn needs Prime Ideals.
Why it matters: It forces the AI to not just translate one sentence, but to figure out: "Do I have the tools to build this? If not, can I find or build the tools first?"

3. The Results: The AI is Stuck in the Mud

The authors tested the smartest AI models available on MathAtlas, and the results were humbling.

The Score: Even the best AI got less than 10% of the theorem statements correct. For definitions, it was around 16%.
The "Depth" Trap: The deeper the "dependency tree" (the more steps you have to go back to find the definitions), the worse the AI performed.
- Analogy: If the AI has to look up 10 previous concepts to translate one sentence, it gets confused and gives up. On the hardest problems (where the dependency tree was deepest), the AI got only 2.6% correct.
The "Mathlib" Effect: The AI did slightly better when the concept it needed to translate was already in a famous library called "Mathlib" (which is like a pre-built toolbox the AI has likely studied). If the concept was new and not in that library, the AI struggled much more.

4. The "Trust" Test (MA-Align)

The paper also created a new test called MA-Align.

The Problem: Sometimes an AI translates a sentence into code that looks perfect and runs without errors, but it actually means something completely different than the original math. It's like translating "The cat sat on the mat" into a computer program that says "The dog ate the pizza." The program works, but it's wrong.
The Test: They asked AI judges to check if the translation was "faithful" (true to the meaning). They found that even the best AI judges were often fooled, especially with complex graduate-level definitions.

The Bottom Line

The paper concludes that while AI is getting good at simple math, it is currently terrible at translating complex, real-world graduate mathematics. The main bottleneck isn't just translating words; it's understanding the massive web of connections between ideas and knowing how to build the necessary "tools" (definitions) before building the "house" (theorems).

MathAtlas is now open for everyone to use as a challenge course to help AI learn how to handle these deep, complex dependencies.

Technical Summary: MathAtlas: A Benchmark for Autoformalization in the Wild

Problem Statement

Current autoformalization benchmarks primarily focus on simpler domains, such as olympiad or undergraduate mathematics, where prerequisite material is limited and often already formalized in libraries like Mathlib. However, graduate-level and research mathematics remain underexplored. In these advanced domains, autoformalization is significantly more complex because:

Prerequisite Depth: Advanced mathematics relies on extensive prerequisite theory, much of which has not yet been formalized. An effective system must not only formalize the target statement but also retrieve or synthesize the necessary dependent theories.
Definition Gap: Existing benchmarks largely ignore the formalization of mathematical definitions. Systems that can only handle theorem statements are limited by the existing formal theory; they cannot formalize theorems involving concepts for which no definitions exist.
Lack of Real-World Data: There is a scarcity of large-scale, real-world benchmarks that capture the "in the wild" complexity of graduate-level mathematics, including rich contextual dependencies and inter-entity relations.

Methodology

Dataset Construction: MathAtlas

The authors introduce MathAtlas, the first large-scale autoformalization benchmark for graduate-level mathematics.

Source Material: The dataset is extracted from 103 graduate-level mathematics textbooks covering 87 distinct fields (e.g., real analysis, differential geometry, category theory, quantum groups).
Scale and Composition: MathAtlas contains approximately 52,052 entities, comprising:
- ~18,000 theorems
- ~10,000 exercises
- ~10,000 definitions
- ~10,000 proofs
- ~5,000 examples
Dependency Graph: A key feature is the enrichment of the dataset with a mathematical dependency graph containing ~178,000 relations. This graph links entities to:
- Object References: Named mathematical concepts (e.g., "Dedekind ring") that require external definitions.
- Entity References: References to other specific theorems or definitions within the text.
- Local Variable References: Symbols used without explicit local introduction.
- Dependency Depth: The maximum height of an entity's dependency tree (average depth of 30, ranging up to 80), which varies significantly by field (e.g., Lie algebra has deeper trees than set theory).

Construction Pipeline:

Conversion: PDFs are converted to Mathematical Markdown (MMD) using Nougat.
Extraction: A few-shot prompted LLM (gpt-oss-120b) extracts entities (definitions, theorems, etc.) and their identifiers.
Reference Extraction: A second model identifies object, entity, and local variable references within the extracted text.
Relation Extraction: A system matches references to target entities (using LeanSearch for Mathlib grounding and LLMs for internal matching) to construct the dependency graph.
Quality Control: Human evaluation on random samples indicates 90.4% validity for entity extraction and a 94.8% F1 score for relation extraction.

Evaluation Metrics

The paper evaluates autoformalization performance using a composite metric called Correctness, which requires a formalization to satisfy two conditions:

Compilation: The generated Lean 4 code must compile successfully.
Semantic Faithfulness: The formal statement must be semantically equivalent to the informal source.
- To measure faithfulness, the authors introduce MA-Align, a binary classification benchmark of 200 entities (statements and definitions) from MathAtlas, annotated as "aligned" or "misaligned."
- They evaluate existing "LLM-as-judge" systems (e.g., CriticLean) and find that while they perform well on prior benchmarks (ConsistencyCheck, CriticLeanBench), their accuracy drops significantly on MA-Align, highlighting a gap in evaluating graduate-level definitions.

Experimental Setup

The authors tested various baselines on MathAtlas, including:

Prompted Models: Zero-shot and few-shot prompting with gpt-oss-20b and gpt-oss-120b.
Fine-Tuned Models: Herald, Kimina, ATLAS-L, Goedel, and ReForm.
Ablation Studies: Testing the impact of local context (previous 500 tokens), engineered prompts, and specific training examples.
MA-Hard Subset: A specific subset of ~700 entities with the deepest dependency trees (average depth >50) to test the limits of current systems.

Key Results

Overall Performance

MathAtlas proves to be an extremely challenging benchmark. Strong baselines achieve low correctness rates:

Statements (Theorems, Examples, Exercises): The best baseline (ReForm 8B) achieves only 9.8% correctness.
Definitions: The best baseline (gpt-oss-120b with few-shot prompting) achieves 16.7% correctness.
Disparity: There is a significant gap between compilation rates and correctness (e.g., Kimina 7B compiles 27.3% of statements but only 2.3% are faithful), indicating that syntactic success does not guarantee semantic accuracy.

Dependency Depth and Difficulty

Performance degrades substantially as dependency depth increases:

On MA-Hard (the 700 entities with the deepest dependency trees), the best model achieves only 2.6% correctness.
Statistical analysis (Kolmogorov–Smirnov test) confirms a significant difference in correctness distributions between shallow and deep dependency trees ( $p < 0.001$ ).

Context and Grounding

Local Context: Adding local context (previous 500 tokens) to the prompt decreased performance for prompted models (dropping from 20.3% to 17.4% on definitions), suggesting current models struggle to integrate long-range context effectively.
Mathlib Grounding: Entities that could be grounded to existing formalizations in Mathlib were significantly more likely to be correctly formalized (27.9% vs. 16.8% for ungrounded definitions), suggesting that LLMs benefit from prior exposure to formalized theory in their training data.

Significance and Contributions

The paper claims the following significance for the field:

First Graduate-Level Benchmark: MathAtlas is the first large-scale, "in the wild" benchmark for graduate-level mathematics, moving beyond the constraints of olympiad and undergraduate datasets.
Dependency-Aware Evaluation: It is the first benchmark to include rich dependency graphs and inter-entity relations, enabling the evaluation of systems that must handle prerequisite theory and definition synthesis.
Highlighting the Definition Gap: The results demonstrate that current state-of-the-art models struggle significantly with definitions and deep dependency chains, revealing a critical bottleneck in scaling autoformalization to advanced mathematics.
Faithfulness Metric Limitations: The introduction of MA-Align and the subsequent experiments show that existing semantic faithfulness metrics (like CriticLean) do not generalize well to graduate-level definitions, necessitating new evaluation approaches.
Community Resource: The authors release MathAtlas, MA-Align, and the experimental code to facilitate future research into dependency-aware autoformalization and the formalization of advanced mathematical concepts.

The authors conclude that while current systems show promise, the gap between existing capabilities and the requirements of graduate-level mathematics remains vast, particularly regarding the synthesis of prerequisite theory and the formalization of definitions.