An Empirical Study of LLM-Generated Specifications for VeriFast
This paper empirically evaluates the effectiveness of ten large language models across eight prompting strategies in generating VeriFast specifications for 303 C functions, revealing that while LLMs preserve functional behavior, they achieve only modest verification success (31.4%) primarily due to errors in domain-specific separation logic knowledge.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very strict, super-smart robot inspector named VeriFast. Its job is to check computer code to make sure it never crashes, never loses data, and behaves exactly as promised. But here's the catch: VeriFast doesn't speak human language. It speaks a very complex, mathematical dialect called Separation Logic. To get VeriFast to do its job, a human has to write a massive "instruction manual" (specifications) explaining exactly how the code handles memory, like a detailed map of a city's traffic rules.
Writing these manuals is incredibly hard and time-consuming. It's like trying to write a legal contract for every single turn a driver makes on a road trip.
The Big Question
The researchers in this paper asked: Can Artificial Intelligence (specifically Large Language Models or LLMs) write these complex instruction manuals for VeriFast?
They treated the LLMs like a new apprentice who knows how to write code but needs to learn VeriFast's strict dialect. They tested 10 different AI models on 303 different computer functions (small pieces of code) to see if the AI could write the manual, and if VeriFast would accept it.
The Experiment: The "Three-Step" Test
The researchers set up a massive test drive:
- The Inputs: They gave the AI three types of "briefings" for each task:
- Natural Language: Just a plain English description (e.g., "This function adds a node to the end of a list").
- Formal Behavior: A mix of code and some math symbols.
- Formal Plus: The code plus a very detailed, almost complete math manual.
- The Prompts: They tried 8 different ways of asking the AI to do the job (like giving it a step-by-step recipe vs. just saying "do it").
- The Models: They tested 10 different AI brains, from GPT-4o to Gemini 2.5 Pro.
The Results: The Good, The Bad, and The "Almost"
The Good News (The Apprentice is Honest):
The AI was surprisingly good at understanding the intent of the code. In over 91% of cases, the AI didn't accidentally change what the code was supposed to do. It didn't try to trick the inspector by making the code do something easier to prove. It stayed true to the original job description.The Bad News (The Apprentice is Clueless about the Rules):
Even though the AI understood the job, it failed to write a manual that VeriFast would accept. Only about 31% of the AI-generated manuals passed the inspection. That's roughly 1 out of every 3 attempts.The "Why" (It's Not Logic, It's Syntax):
When the AI failed, it wasn't usually because it was "stupid" or couldn't do the math. It failed because it didn't know the specific grammar and rules of VeriFast.- Analogy: Imagine the AI is a brilliant chef who knows how to cook a perfect steak. But VeriFast is a health inspector who only accepts recipes written in a specific, obscure dialect of French. The chef keeps writing the recipe in English or Spanish. The food is great, but the inspector rejects it because the format is wrong.
- The paper found that 94% of the errors were about these specific rules (like forgetting to "open" or "close" a memory block, or missing a specific keyword), not about the logic of the program itself.
Who Won?
- The Best AI: Gemini 2.5 Pro was the clear winner. It made fewer mistakes and passed more inspections than the others.
- The Best Input: When the researchers gave the AI a "Formal Plus" briefing (a very detailed starting point), the success rate went up. When they just gave it a plain English description, the AI struggled the most.
The Takeaway
The paper concludes that while AI is great at understanding what code should do, it is currently not great at writing the specific, rigid instructions required by advanced verification tools like VeriFast.
The AI isn't failing because it can't reason; it's failing because it doesn't know the "dialect" of the tool. To fix this, the researchers suggest that we need to either:
- Teach the AI the specific rules of VeriFast better.
- Give the AI better feedback when it makes a mistake (like a teacher correcting the grammar, not just saying "wrong").
- Use a mix of AI and traditional tools to help fill in the gaps.
In short: The AI apprentice is talented and honest, but it needs a lot more training on the specific "language" of the robot inspector before it can work alone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.