MathForm: Scaling Mathematical Autoformalization with Knowledge Retrieval and Verification-Guided Refinement

Imagine you are trying to teach a brilliant but slightly clumsy robot how to speak the language of pure logic. This robot, a Large Language Model (LLM), is amazing at reading stories, writing poems, and solving math puzzles in plain English. But there's a catch: to prove a math theorem with absolute certainty, you can't just use words; you need to write it in a super-strict computer language called Lean 4. Think of Lean 4 as a high-security vault where every single word, symbol, and rule must be perfect, or the vault door won't open. The problem is that while the robot knows math, it doesn't know the specific "rulebook" (called Mathlib) that Lean 4 uses. It's like asking a chef who knows how to cook a perfect steak to follow a recipe written in a language they've never seen, using ingredients they don't know the names of. They might guess, but they'll likely get it wrong.

This is where "autoformalization" comes in: the art of translating human math into this strict computer code. For a long time, researchers tried to just ask the robot to "translate" the math, hoping it would remember the rules from its training. But the robot kept making mistakes, like using the wrong ingredients or forgetting a crucial step, because it was trying to rely on its memory alone. The new paper, MathForm, argues that this "guess and hope" approach is broken. Instead, they built a system where the robot is allowed to look up the rulebook before it starts writing, and if it makes a mistake, a strict editor doesn't just throw the work away—they tell the robot exactly what went wrong and let it try again until it gets it right.

The researchers behind MathForm realized that to get a robot to speak the language of formal math perfectly, you can't just let it write a draft and hope for the best. They built a three-step assembly line to fix the robot's workflow. First, before the robot writes a single line of code, a "Researcher" agent scans the massive Mathlib library to find the exact definitions and rules the robot needs for that specific problem. It's like giving the chef the specific cookbook page for "Steak" before they even pick up a knife. Second, the robot writes its code, and then a "Inspector" checks it. If the code has a syntax error (like a missing comma), the Inspector points it out. If the code compiles but means the wrong thing (like saying "all numbers" when the problem meant "only positive numbers"), the Inspector explains the semantic error. Third, instead of giving up, the robot uses this feedback to rewrite its code. It keeps looping through this "write-check-fix" cycle until the code is perfect.

Using this clever loop, the team created a massive new dataset called FormalVerse, containing about 367,000 verified math examples. They then trained a new model, MathForm-8B, on this data. The results were surprising: this relatively small model (8 billion parameters) became better at formalizing math than much larger, specialized models (32 billion parameters) that relied on the old "guess and hope" methods. On six different difficult math tests, MathForm-8B passed the strict "Consistency Check" (meaning the code actually meant what the human problem said) about 72.4% of the time, beating the previous best models. Even on the hardest, most abstract algebra problems, it significantly outperformed its bigger rivals. The paper suggests that by giving the model the right tools to look up information and the chance to learn from its mistakes, you don't need a giant brain to be a math genius; you just need a smart workflow.