Technical Summary: LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in "informal" mathematical reasoning (natural language), yet they struggle to generate mechanically verifiable proofs in formal languages like Lean. While specialized models fine-tuned on formal corpora have shown promise, general-purpose foundation models often underperform in one-shot formal theorem proving. Furthermore, existing benchmarks (e.g., MiniF2F, PutnamBench) are becoming saturated, and current specialized systems often rely on closed-source components or fail to scale to highly non-routine, multi-step proofs required by advanced competitions like the International Mathematical Olympiad (IMO).

The core challenge is bridging the gap between the strong informal reasoning of general LLMs and the rigorous, machine-checkable requirements of formal verification without relying solely on specialized fine-tuning.

2. Methodology: LEAP Framework

The authors introduce LEAP (LLM-in-Lean Environment Agentic Prover), an agentic framework designed to enable general-purpose foundation models to achieve state-of-the-art performance in automated formal theorem proving. LEAP does not rely on specialized fine-tuned prover models but instead leverages the complementary skills of general LLMs: informal reasoning, instruction following, tool use, and iterative self-refinement.

2.1. Blueprint-Driven Hierarchical Decomposition

Inspired by human mathematical workflows and the Lean Blueprint tool, LEAP decomposes complex proofs into manageable units. The system operates on an AND-OR Directed Acyclic Graph (DAG):

• OR Nodes: Represent open goals or lemma statements that can be resolved by any valid proof strategy.
• AND Nodes: Represent candidate decompositions where the parent goal is considered proved only if all constituent subgoals are proved.

2.2. Workflow and Iterative Refinement

LEAP follows a multi-stage workflow (illustrated in Figure 1 of the paper):

1. Direct Formalization Attempt: The agent first attempts to generate a direct informal proof, translate it to Lean, and verify it with the Lean compiler.
2. Decomposition (Blueprinting): If direct proving fails, LEAP drafts an informal blueprint proposing intermediate lemmas. This blueprint is translated into a formal proof sketch.
   • The sketch proves the current goal assuming the proposed lemmas (using sorry placeholders only for the new lemmas).
   • If the sketch is syntactically valid and type-correct, it is added as an AND node, and the proposed lemmas become child OR nodes.
3. Verification-Guided Search:
   • Compiler Feedback: The Lean compiler validates syntax and types.
   • LLM Reviewer: A critical component where an LLM reviewer assesses the quality of the decomposition. It filters out unproductive branches (e.g., subgoals that are as hard as the original problem or syntactically identical to the parent) to prevent search explosion.
4. Hierarchical Memoization: The DAG structure allows for the storage and reuse of intermediate lemmas across different branches, preventing redundant derivations and supporting "anticipatory lemma planning" (proposing lemmas not immediately needed but useful later).

3. Key Contributions

3.1. The LEAP Agentic Framework

The paper demonstrates that state-of-the-art formal theorem proving can be achieved using only general-purpose LLMs when coupled with a structured agentic workflow. This challenges the prevailing assumption that specialized fine-tuning is indispensable for rigorous formal tasks. The framework codifies a human-like workflow: high-level blueprint sketching followed by low-level formal generation and iterative compiler feedback.

3.2. Lean-IMO-Bench Dataset

To evaluate progress beyond saturated benchmarks, the authors introduce Lean-IMO-Bench, a new dataset formalizing 60 problems from the IMO-Bench (Luong et al., 2025) into Lean.

• Structure: Divided into a "Basic" set (pre-IMO to IMO-Medium) and an "Advanced" set (up to IMO-Hard).
• Characteristics: Problems feature short statements but require highly non-routine, multi-step, and structurally intricate proofs.
• Verification: All 60 problem statements were manually formalized and verified by Lean experts to ensure accuracy.

3.3. Empirical Results and Analysis

The paper provides extensive empirical evidence across two major benchmarks:

• Putnam 2025: LEAP solved 100% of the 12 problems (12/12), matching the performance of frontier closed-source systems (Aristotle, Numina) and significantly outperforming direct baselines (0% solve rate for general LLMs and specialized models in one-shot settings).
• Lean-IMO-Bench: LEAP increased the one-shot formal solve rate of general-purpose LLMs from <10% to 70%. It notably surpassed specialized ATP models (5%) and the gold-medal-caliber system Aristotle (48%). LEAP achieved 100% solve rates in Algebra and Number Theory across both difficulty tiers.

3.4. Ablation Studies

The authors conducted ablation studies to isolate the contributions of LEAP's components:

• Iterative vs. One-Shot: General LLMs (Gemini-3.1-Pro) improved significantly (20% to 36.6%) with iterative compiler feedback, whereas specialized models (Goedel-Prover-V2) did not benefit as much, highlighting the importance of general reasoning capabilities.
• DAG vs. Tree: Removing the global DAG memoization (using a naive tree structure) reduced performance from 83.3% to 73.3% on the Basic set, confirming the value of lemma reuse and graph context.
• LLM Reviewer: Removing the LLM-based decomposition reviewer caused the agent to fail on complex problems (e.g., Putnam A5) due to unproductive cyclic decompositions, proving the necessity of semantic search filtering.

4. Significance and Claims

The paper claims that the primary bottleneck for general-purpose LLMs in formal mathematics is not language comprehension but the lack of structured, iterative interaction with the proof environment. By scaffolding general LLMs with an agentic framework that includes hierarchical decomposition, compiler feedback, and semantic review, the system can bridge the gap between informal reasoning and formal verification.

The authors demonstrate that general foundation models, when placed in the correct agentic environment, can progress from near-zero formal math performance to solving highly complex, open research problems. This is evidenced by LEAP's autonomous formalization of a key subproblem in Knuth's Hamiltonian decomposition of even-order Cayley graphs, a complex combinatorial challenge requiring over 5,000 lines of verified Lean code.

The work suggests a paradigm shift: rather than relying solely on small, specialized models trained on formal corpora, the field can leverage the extensive knowledge, instruction-following, and self-correction capabilities of general foundation models to achieve competitive, and in some cases superior, results in formal theorem proving.