LeanTutor: Towards a Verified AI Mathematical Proof Tutor

This paper presents LeanTutor, a proof-of-concept AI system that combines Large Language Models with the Lean theorem prover to provide verified mathematical proof tutoring, evaluated using the newly introduced PeanoBench dataset.

The Gist

Imagine you are trying to learn how to write a perfect, unbreakable legal contract. You have two potential teachers, but both have a major flaw:

1. The Chatty Friend (The LLM): This teacher is incredibly friendly, speaks your language fluently, and can explain complex ideas with great stories. However, they are a bit of a daydreamer. They often make up facts, get the details wrong, and their contracts are full of hidden loopholes. They think they are right, but they aren't always.
2. The Robot Judge (The Theorem Prover): This teacher is a machine that never lies. If they say a contract is valid, it is 100% mathematically guaranteed to be correct. But here's the catch: they only speak in a secret, robotic code that is incredibly hard for humans to understand. If you ask them a question in plain English, they just stare at you blankly.

The Problem: Students need the friendliness of the Chatty Friend to learn, but they need the absolute accuracy of the Robot Judge to be safe.

The Solution: LeanTutor
The paper introduces LeanTutor, a new system that acts like a super-powered translator and coach, combining the best of both worlds. Think of it as a bilingual tour guide who speaks both "Human" and "Robot."

LeanTutor works like a three-person dream team:

• The Translator (Autoformalizer/Proof-Checker): When you write a math idea in plain English, this module instantly translates it into the Robot Judge's secret code. It then asks the Robot, "Is this actually true?" If the Robot says "No," the Translator tells you exactly where your logic broke.
• The Coach (Next-Step Generator): Instead of just saying "Wrong," this module acts like a helpful hiking guide. If you are stuck on a mountain path, it doesn't carry you up; it points to the next safe step you should take to keep climbing. It suggests the next logical move in the proof.
• The Storyteller (Natural Language Feedback Generator): Once the Robot Judge has checked the math, this module takes the cold, robotic "Error Code 404" and turns it into a warm, encouraging explanation in plain English. It tells you why you made a mistake and how to fix it, just like a human tutor would.

The Test Drive: PeanoBench
To see if this team actually works, the authors built a training ground called PeanoBench. Imagine a gym with 371 specific math puzzles (based on the rules of counting numbers). They created a dataset where every puzzle has both a "human story" version and a "robot code" version. This allows them to test if LeanTutor can successfully guide a student from the story to the correct robot code without making mistakes.

In a Nutshell
LeanTutor is an AI tutor that uses the brain of a human-friendly chatbot to talk to you, but uses the soul of a strict math robot to check your work. It ensures that when you learn math, you are learning the truth, not just a convincing story.

Technical Summary

Based on the abstract provided, here is a detailed technical summary of the paper "LeanTutor: Towards a Verified AI Mathematical Proof Tutor":

1. Problem Statement

The paper addresses a critical gap in AI-assisted mathematics education. While Large Language Models (LLMs) excel at natural language communication and can explain concepts fluently, they are inherently error-prone and cannot guarantee the logical correctness of mathematical proofs. Conversely, formal theorem provers like Lean offer provably correct verification, ensuring mathematical rigor, but they possess a steep learning curve that makes them inaccessible to most students. The core challenge is creating a system that combines the accessibility of LLMs with the rigorous correctness guarantees of formal theorem provers.

2. Methodology

The authors propose LeanTutor, a proof-of-concept system designed to bridge the gap between natural language interaction and formal verification. The system integrates the complementary strengths of LLMs and theorem provers through a three-module architecture:

• Module (i): Autoformalizer/Proof-Checker
  • This module translates natural language mathematical statements and student attempts into the formal language of Lean.
  • It acts as the "ground truth" validator, using the theorem prover to verify whether a specific step or proof is logically sound.
• Module (ii): Next-Step Generator
  • Leveraging the LLM, this module suggests the next logical step in a proof based on the current state.
  • Crucially, these suggestions are constrained or guided by the formal verification capabilities of the system to ensure they are viable within the Lean framework.
• Module (iii): Natural Language Feedback Generator
  • Once the formal system validates (or invalidates) a step, this module translates the formal feedback into intuitive, natural language explanations for the student.
  • This ensures the student receives pedagogical guidance that is both easy to understand and mathematically verified.

3. Key Contributions

• LeanTutor System: The development of a novel, modular architecture that successfully couples LLMs with the Lean theorem prover to create a "verified" tutoring environment.
• PeanoBench Dataset: The introduction of a new benchmark dataset specifically designed to evaluate such systems. It contains 371 proofs in Peano Arithmetic, featuring parallel representations in both human-written natural language and formal Lean language.
• Data Source: The dataset is derived from the Natural Numbers Game, a popular interactive tutorial for Lean, ensuring high-quality, pedagogically relevant data.

4. Results

• Note: The provided abstract does not contain specific quantitative metrics (e.g., accuracy percentages or pass rates).
• However, the paper establishes the feasibility of the approach by successfully constructing the proof-of-concept system. The creation of PeanoBench serves as a foundational result, providing the necessary infrastructure to quantitatively evaluate the performance of AI tutors in formal mathematics, a domain previously lacking standardized benchmarks.

5. Significance

• Bridging Rigor and Accessibility: LeanTutor represents a significant step toward making formal verification accessible to learners. It allows students to interact with mathematics in natural language while ensuring that the underlying logic is mathematically irrefutable.
• Educational Impact: By automating the translation between natural language and formal logic, the system lowers the barrier to entry for learning theorem proving, potentially transforming how advanced mathematics is taught.
• Future Research: The introduction of PeanoBench sets a standard for future research in AI-assisted formal mathematics, enabling the community to benchmark and improve upon verified AI tutors systematically.