Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

Imagine you are trying to teach a very smart, well-read robot how to write a strict, mathematical recipe for a complex machine. This machine is a "distributed system" (like the cloud servers running Amazon or Microsoft), and the recipe is written in a special language called TLA+.

This language is like a high-stakes puzzle. If you miss one tiny symbol or get the logic slightly wrong, the machine might work in the recipe but crash in real life. The problem is that writing these recipes by hand is hard and slow. So, researchers asked: Can we just ask a modern AI (a Large Language Model, or LLM) to write these recipes for us?

This paper is the first big report card on that question. Here is what they found, explained simply:

1. The "Grammar vs. Meaning" Gap

The researchers asked 30 different AIs to write these TLA+ recipes based on plain English descriptions.

• The Good News (Grammar): About 26% of the time, the AI wrote a recipe that looked correct on the surface. The "spell-checker" (called SANY) said, "Okay, the words and symbols are in the right order."
• The Bad News (Meaning): However, when they actually ran the recipe through a "logic tester" (called TLC) to see if it actually worked, only 8.6% of the time did it pass.

The Analogy: Imagine asking a student to write a legal contract. The student uses perfect spelling and grammar (26% success), but the contract they wrote actually says the opposite of what was intended, or leaves out a crucial clause, making it legally useless (only 8.6% success). The AI is great at mimicking the look of the language but often fails to understand the logic behind it.

2. Bigger Isn't Always Better

Usually, we assume a bigger, more powerful AI will do a better job. But in this study, that wasn't true.

• The Surprise: A smaller AI model (DeepSeek r1:8b) actually did a much better job than its massive "big brother" (DeepSeek r1:70b).
• Why? The smaller model was trained specifically to "think step-by-step" (like a math student showing their work), while the bigger model was trained on so much general internet data that it got confused by the strict rules of TLA+. It's like a specialized chef who knows exactly how to bake a soufflé, versus a generalist who knows how to cook everything but might overthink the specific recipe.

3. "Code Experts" Failed

The researchers tested AIs that are famous for writing computer code (like Python or Java). Surprisingly, these "code experts" did worse than the general-purpose AIs.

• The Reason: These models are so used to writing code with semicolons (;) and curly braces ({}) that they kept accidentally putting those into the TLA+ recipe. Since TLA+ doesn't use those symbols, the recipe broke immediately. It's like a carpenter trying to fix a watch but accidentally trying to use a hammer because that's what they use for everything else.

4. The "Step-by-Step" Trick Worked Best

The researchers tried four different ways to ask the AI for help. The most successful method was called "Progressive Prompting."

• How it worked: Instead of asking the AI to write the whole recipe at once, they asked it to build it piece by piece: "First, write the title. Now, write the variables. Now, write the rules."
• The Result: This was the only method that produced any fully working recipes (the 8.6% success rate). It's like building a house: if you try to build the roof, walls, and foundation all in one giant leap, you'll likely fail. But if you build them one room at a time, you have a better chance of success.

5. The AI's "Hallucinations"

The paper found five specific ways the AI kept making the same mistakes, which they call "hallucinations":

1. Wrong Symbols: Using fancy math symbols (like ∧) instead of the plain text symbols TLA+ requires (like /\).
2. Language Mixing: Accidentally adding semicolons or backticks from other programming languages.
3. Thinking Out Loud: The AI sometimes pasted its own "thinking process" (like ...) directly into the final recipe, which broke it.
4. Wrong Length: Sometimes the AI wrote a recipe that was 9 times too long, or sometimes it wrote almost nothing at all.
5. Broken Structure: Missing the "end" markers of the recipe, leaving the document unfinished.

The Bottom Line

The paper concludes that current AI cannot yet write reliable TLA+ specifications on its own. While it can mimic the look of the language, it still makes too many logical errors to be trusted without a human expert checking every single line.

The researchers suggest that to fix this, we need to:

• Use the "step-by-step" prompting method.
• Use smaller, reasoning-focused models rather than massive general ones.
• Build tools that automatically fix the common mistakes (like removing the wrong symbols) before the AI even tries to run the recipe.

Until then, writing these critical system recipes remains a job for human experts, with AI acting as a helpful but error-prone assistant.