🤖 AI

Aria: An Agent For Retrieval and Iterative Auto-Formalization via Dependency Graph

The paper introduces Aria, a retrieval-based agent that employs a two-phase Graph-of-Thought process and a definition-grounded scorer to achieve state-of-the-art accuracy in conjecture-level auto-formalization of research mathematics in Lean, effectively overcoming common LLM limitations like hallucinations and semantic mismatches.

Original authors: Hanyu Wang, Ruohan Xie, Yutong Wang, Guoxiong Gao, Xintao Yu, Bin Dong

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanyu Wang, Ruohan Xie, Yutong Wang, Guoxiong Gao, Xintao Yu, Bin Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Lost in Translation" Gap

Imagine you have a brilliant mathematician who speaks "Human Math" (natural language, like English) and a super-strict computer that only speaks "Formal Math" (a rigid programming language called Lean). The computer is incredibly powerful; it can prove theorems without making mistakes. But it has a huge problem: it doesn't understand human math unless it is translated perfectly.

If you translate a math sentence even slightly wrong, the computer rejects it. Current AI models (LLMs) are like enthusiastic but inexperienced translators. They often:

Hallucinate: They invent words or rules that don't exist in the computer's dictionary.
Get the meaning wrong: They use the right words but in the wrong order, changing the meaning entirely.
Give up on hard stuff: When faced with a brand-new, complex research problem, they freeze because they don't have a pre-written answer in their memory.

The Solution: Meet Aria

The authors built a new AI agent named Aria. Think of Aria not as a translator, but as a Master Architect who builds a bridge between Human Math and Computer Math.

Aria doesn't just guess the translation. It uses a three-step process to ensure the bridge is solid:

1. The "Dependency Map" (Graph-of-Thought)

The Analogy: Imagine you are trying to build a skyscraper. You can't just pour concrete for the top floor; you need the foundation, the steel beams, and the plumbing first.
How Aria does it: Instead of trying to translate the whole math problem at once, Aria breaks it down. It draws a map (a graph) showing how every concept depends on others.

Example: To understand "Cohen-Macaulay Module," you first need to understand "Noetherian Ring," which needs "Ideal," which needs "Ring."
Aria builds the bridge from the bottom up, ensuring every single brick is laid correctly before moving to the next level.

2. The "Library Detective" (Retrieval-Augmented Generation)

The Analogy: Imagine a student taking a test who is allowed to use a library but isn't allowed to memorize the books. If the student tries to make up a rule, they get caught.
How Aria does it: Math libraries (like Mathlib) are constantly updated with new rules. Old AI models rely on what they memorized years ago, which is often outdated or wrong.

Aria acts like a detective. Before it writes a single line of code, it searches the current library to find the exact, official definition of the terms it needs.
If the library doesn't have a definition for a new concept, Aria knows it has to invent one from scratch, but it does so carefully, checking its work against the rules it just found.

3. The "Self-Correction Loop" (Iterative Reflection)

The Analogy: Think of a sculptor chipping away at a statue. They don't just carve once and hope for the best. They chip, step back, look at it, see a flaw, and chip again.
How Aria does it:

Aria writes a piece of code.
It runs it through the computer compiler (the strict judge).
If the computer says "Error," Aria doesn't panic. It reads the error, figures out what went wrong, and tries again.
It repeats this "try-fail-fix" cycle until the code compiles perfectly.

The "Truth Detector": AriaScorer

Even if the code compiles (runs without crashing), it might still mean the wrong thing. This is like a sentence that is grammatically perfect but says, "The sky is green."

The authors built a special tool called AriaScorer to check the meaning.

The Old Way: Previous tools just compared the words. If the human said "Ring" and the code said "Ring," they thought it was a match.
The Aria Way: AriaScorer is a deep-dive investigator. It looks up the actual definition of "Ring" in the computer's library and compares that deep definition to the human's intent.
It catches subtle tricks, like when the AI swaps the order of ingredients in a recipe. It ensures the computer's version is mathematically identical to the human's idea, not just similar-looking.

The Results: How Good is Aria?

The team tested Aria on three levels of difficulty:

Undergraduate Math (ProofNet): Aria got 68.5% of the hard translations right, beating all previous models.
PhD Level Math (FATE-X): This is where other models usually fail. Aria got 44.0% right, while the next best model only got 24.0%.
Real Research Conjectures (The "Impossible" Test): The team gave Aria 14 brand-new, unsolved math problems from real mathematicians.
- Other AI models: 0% success. They couldn't even start.
- Aria: 42.9% success. It successfully translated nearly half of these brand-new, never-seen-before problems into a format the computer could understand.

Summary

Aria is a system that stops AI from "making things up" when doing advanced math. Instead of guessing, it:

Maps the dependencies (like a blueprint).
Searches the official library for current rules (like a detective).
Iterates and fixes its own mistakes (like a sculptor).
Verifies the deep meaning, not just the surface words (like a truth detector).

This allows it to handle complex, research-level math problems that previous AI systems simply couldn't touch.

Technical Summary: ARIA – An Agent for Retrieval and Iterative Auto-Formalization via Dependency Graph

1. Problem Statement

The accurate auto-formalization of theorem statements—translating informal mathematical text into formal languages like Lean 4—is a critical bottleneck for advancing automated discovery and verification in research-level mathematics. While Large Language Models (LLMs) have shown promise in undergraduate-level tasks, they struggle significantly with conjecture-level statements due to three primary failure modes:

Hallucinations and Semantic Mismatches: LLMs often generate code using functions or definitions that do not exist in the library (Mathlib) or misinterpret the semantic intent of mathematical concepts.
Inability to Synthesize New Definitions: Research-level mathematics frequently requires defining novel objects not present in existing libraries. Static, one-pass generation methods fail because they cannot spontaneously synthesize these definitions.
Semantic Verification Limitations: Existing semantic checkers often rely on superficial textual similarity or back-translation, failing to detect subtle discrepancies between informal concepts and their formal Lean counterparts (e.g., incorrect type coercions or reversed parameter orders).

These issues lead to unfaithful translations that derail large-scale data generation pipelines and prevent autonomous exploration of open mathematical problems.

2. Methodology

The paper introduces ARIA (Agent for Retrieval and Iterative Auto-Formalization), a system designed to emulate human expert reasoning through a two-phase Graph-of-Thought (GoT) process. The pipeline is structured as follows:

2.1 The Graph-of-Thought (GoT) Pipeline

Unlike direct single-step generation, ARIA decomposes the formalization task into a structured dependency graph:

Phase 1: GoT Decomposition (Top-Down): The agent takes an informal statement and recursively decomposes it into a dependency graph of concepts. Each node represents a definition, structure, or class.
- Grounding: For each concept, the agent queries LeanSearch (a retrieval engine indexing the latest Mathlib) to find canonical definitions.
- Reasoning: An LLM analyzes retrieved candidates to identify the best canonical match. If no match exists (the concept is new), the node is marked for synthesis.
Phase 2: GoT Synthesis (Bottom-Up): The agent synthesizes formal definitions starting from the leaf nodes (grounded concepts) up to the root (the main theorem).
- Compiler-in-the-Loop: At every synthesis step, the generated code is immediately compiled. If compilation fails, the error message is fed back to the LLM for refinement (self-reflection).
- Synthesis of New Concepts: For ungrounded nodes, the agent generates new definitions using the verified code of their dependencies as context.

2.2 AriaScorer: Term-Level Grounded Semantic Checker

To ensure semantic correctness beyond syntactic compilation, the authors introduce AriaScorer.

Term-Level Retrieval: Before evaluating a formal statement, AriaScorer uses a static analyzer (jixia) to extract all Lean terms referenced in the code. It queries a curated Mathlib dataset to retrieve the authoritative definitions, types, and informal descriptions of these terms.
Subtask Decomposition: The informal statement is decomposed into atomic assumptions and conclusions.
Grounded Evaluation: The LLM evaluates the match between the informal subtasks and the formal code, but crucially, it is provided with the retrieved formal definitions as context. This allows the checker to reason about the true semantics of the Lean terms rather than relying on surface-level text similarity.
Scoring: Subtasks are labeled as "Perfectly Match," "Minor Inconsistency," or "Major Inconsistency." These are aggregated via a fuzzy integral into a final score (0 to 1), which is thresholded to make a binary accept/reject decision.

3. Key Contributions

Aria Agent: A novel auto-formalization agent that integrates Retrieval-Augmented Generation (RAG), Graph-of-Thought planning, and compiler-guided self-reflection. It is the first system capable of autonomously synthesizing complex, novel definitions required for conjecture-level formalization.
AriaScorer: A term-level grounded semantic checker that retrieves authoritative Mathlib definitions to verify the mathematical correctness of formalizations, significantly reducing false positives caused by semantic hallucinations.
State-of-the-Art Performance: The system achieves significant improvements over existing methods across multiple benchmarks, particularly on research-level problems where previous models fail completely.

4. Experimental Results

The authors evaluated ARIA on diverse benchmarks, measuring Final Accuracy (the proportion of statements passing both compilation and semantic checks).

ProofNet (Undergraduate Level): ARIA achieved a 91.6% compilation success rate and 68.5% final accuracy, surpassing previous methods.
FATE-H (Algebra Final Exams): ARIA achieved 71.0% final accuracy, outperforming the best baseline (Goedel-V2 pass@128) which scored 43.0%.
FATE-X (PhD/Research Level): ARIA achieved 44.0% final accuracy, compared to 24.0% for the best baseline.
Homological Conjectures (Real-World Research): On a dataset of 14 challenging conjectures from commutative algebra, ARIA achieved a 42.9% success rate, while all other baseline models (including Goedel-V2 and Gemini-2.5-Pro) scored 0%.

Ablation Studies:

Removing the Reflection mechanism caused performance to collapse (0% success on Conjectures).
Removing the GoT planner reduced success on Conjectures from 6/14 to 1/14.
Disabling the RAG module resulted in 0% success on Conjectures, confirming that static pre-trained knowledge is insufficient for research-level definitions.

Semantic Checker Validation:
On the FATE-X benchmark, AriaScorer achieved 89.9% accuracy (at threshold $\alpha=0$ ) and 95.5% precision (at $\alpha=0.9$ ), significantly outperforming LeanScorer and Back-Translation baselines. Case studies demonstrated its ability to detect "Definition Discrepancies" (e.g., subtle differences in multiplication rules for quaternion algebras) and suppress hallucinations by grounding the evaluation in actual library definitions.

5. Significance and Claims

The paper claims that ARIA represents a foundational step toward autonomous mathematical discovery. By addressing the "foundational bottlenecks" of hallucination and the inability to synthesize new definitions, ARIA enables the reliable formalization of conjecture-level problems.

The authors emphasize that accurate statement formalization is a prerequisite for theorem proving. The success of ARIA on the Homological Conjectures dataset demonstrates a unique capability to handle active research problems that were previously inaccessible to automated systems. The integration of term-level grounding in semantic checking (AriaScorer) is presented as a necessary evolution beyond textual similarity metrics, providing a rigorous standard for verifying the mathematical fidelity of AI-generated formalizations.

The work suggests that future automated mathematical proof systems must rely on such structured, retrieval-augmented, and self-reflective agents to navigate the complexity of modern mathematical libraries and research frontiers.