VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation
This paper introduces VeriContest, a comprehensive benchmark of 946 competitive programming problems in Rust with Verus that pairs natural language descriptions with expert-validated formal specifications and machine-checkable proofs, revealing a significant performance gap between current models' coding abilities and their capacity for verifiable code generation.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a brilliant but inexperienced architect to build a house.
In the world of standard coding benchmarks, you give the architect a simple description: "Build a house with three bedrooms and a kitchen." The architect draws up blueprints, builds the house, and you check if the doors open and the lights work. If they do, the architect gets a passing grade. This is like current AI models writing code: they are great at making things that look and act right.
But what if you need a house that is mathematically guaranteed to never collapse, even in a hurricane? You can't just check the lights; you need a formal proof that the structure is sound. This is where the paper "VeriContest" comes in.
The Problem: "It Works, But Is It True?"
Current AI models are like those talented architects who can build a house that passes a visual inspection. However, they often skip the rigorous engineering math. They might build a house that looks fine but has a hidden flaw in the foundation that only shows up under specific stress.
The authors of this paper argue that we need a new way to test AI. Instead of just asking, "Does the code run?" we need to ask, "Can you prove, with mathematical certainty, that this code does exactly what it's supposed to do, and nothing else?"
The Solution: VeriContest
The team created a massive "exam" called VeriContest. Think of it as a high-stakes competition for AI architects, but with three strict rules:
- The Blueprint (Specification): The AI must first write a mathematical contract. This isn't just a description; it's a rigid set of rules defining exactly what the input is and what the output must be.
- The Construction (Code): The AI must write the actual code (in the Rust programming language) that follows those rules.
- The Engineering Proof (Verification): The AI must provide a mathematical proof that the code cannot fail. It's like showing the math that proves the roof won't fall off, rather than just hoping it doesn't.
They tested this on 946 difficult puzzles taken from famous coding competitions (LeetCode and Codeforces). These aren't simple "Hello World" tasks; they are complex logic problems involving things like finding patterns in data or optimizing routes.
The Construction Process
Building this exam was hard. The team didn't just ask an AI to make the questions; they built it in three phases:
- Phase 1 (The Seed): Human experts manually wrote 91 perfect examples with flawless proofs.
- Phase 2 (The Expansion): They used an AI assistant to generate more problems, but human experts acted as "editors," checking every single one to ensure the math was correct.
- Phase 3 (The Stress Test): They created "negative test cases"—scenarios designed to trick the AI. If the AI's proof was incomplete, these trick questions would expose the flaw.
The Results: A Huge Gap
When they ran the world's smartest AI models through this exam, the results were surprising and stark.
- The "Normal" Test: When asked just to write code from a description (no proofs required), the best AI got it right 92% of the time. It's a master builder.
- The "Blueprint" Test: When asked to write the mathematical contract (specification), the score dropped to 48%. The AI struggled to define the rules precisely.
- The "Proof" Test: When asked to provide the mathematical proof that the code works, the score plummeted to 14%. The AI couldn't do the heavy lifting of the math.
- The "Full Exam" (End-to-End): When asked to do all three steps at once (Blueprint + Code + Proof), the best AI only succeeded 5.3% of the time.
The Analogy: The "Perfect House"
Imagine the AI is a chef.
- Standard Coding: You ask for a burger. The chef makes a burger that tastes good. You eat it. Success!
- Verifiable Coding: You ask for a burger, but you also demand a certificate proving that the meat came from a specific farm, the bun was baked at exactly 350 degrees, and the burger contains no hidden allergens. The chef can make the burger, but they are terrible at writing the certificate or proving the math behind the cooking process.
The Conclusion
The paper concludes that while AI is getting very good at "guessing" the right code to make a program run, it is still very bad at proving that the code is correct. The biggest bottleneck isn't writing the code; it's writing the formal rules and the mathematical proofs that guarantee the code is safe.
VeriContest is now a tool for researchers to measure exactly how far AI has to go before it can be trusted to build software that is mathematically guaranteed to be bug-free.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.