Combining Tests and Proofs for Better Software Verification
This paper proposes a complementary approach to software verification that integrates testing and formal proving by using Design by Contract and SMT-based counterexample generation to automate test suite creation, regression testing, and program repair.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building a complex Lego castle. To make sure it’s sturdy, you have two ways to check your work:
- The "Stress Test" (Testing): You poke the castle, shake the table, or drop a heavy book near it to see if it falls apart. It’s practical, but you can’t shake the table in every possible way, so you might miss a weak spot.
- The "Blueprint Audit" (Proving): You sit down with the instruction manual and use math to prove that, based on the laws of physics, the castle should never fall. It’s perfect in theory, but if the math says "it might fail," it doesn't tell you where or why. It just gives you a frustrating error message.
For a long time, engineers have argued about which is better. The authors of this paper say: "Why choose? Let's make them talk to each other."
Here is how they combine these two worlds using three clever "superpowers":
1. The "Translator" (Proof2Test)
Imagine you’re studying for a math exam. You solve a problem, and the teacher just circles it in red and says, "Wrong." You’d be frustrated! You’d want the teacher to say, "You got it wrong because you forgot to carry the one when you reached step three."
Currently, "Proving" tools are like that unhelpful teacher. They tell you a program is wrong, but they don't tell you how. This paper introduces a tool that acts as a translator. It takes that abstract "Wrong" and turns it into a concrete example. Instead of saying "The math is broken," it says, "If you put the number 5 into this slot, the machine breaks." It turns a confusing math error into a real-world "stress test" you can actually see and fix.
2. The "Robot Mechanic" (Proof2Fix)
Imagine your car breaks down. Usually, a mechanic has to guess which part is broken, try a new part, and see if it works. Sometimes they make it worse!
This paper proposes a "Robot Mechanic." Because the "Proving" tool already knows exactly why the math failed, it can suggest a fix. But here’s the magic: once the robot suggests a fix, it doesn't just hope it works; it runs the math again to guarantee the fix is perfect. It’s like a mechanic who can see into the future to ensure the car will never break in that specific way again.
3. The "Controlled Sabotage" (Seeding Contradiction)
This is the most creative part. Usually, we write tests to see if a working program stays working. But how do you know if your tests are actually good? How do you know if you've tested every single corner of the castle?
The authors suggest a "spy" technique: Deliberate Sabotage.
They take a perfectly good program and intentionally "plant" tiny, invisible bugs in every single corner. Then, they ask their math tools to find those bugs. By finding the bugs they planted, the tools automatically generate a "Master Checklist" of tests.
It’s like a security company testing a bank vault by intentionally leaving the back door unlocked, then seeing if their sensors catch it. If the sensors catch the "fake" break-in, you know your security system is actually working. This creates a massive, high-quality testing suite that covers every possible nook and cranny of the software.
The Big Picture
Instead of seeing Testing (the hands-on shaker) and Proving (the brainy mathematician) as enemies, this paper turns them into a team. The mathematician finds the flaws, the shaker proves they are real, and together they build software that is much harder to break.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.