Hardening vibe-coded web applications with an automated security-audit skill: a controlled comparison of two builds of the same app
This paper demonstrates that integrating an automated security-audit skill into the "vibe coding" process drastically improves the security of AI-generated web applications, reducing unmitigated risk by 99% and eliminating all high-severity vulnerabilities in a controlled comparison of two builds of the same e-commerce demo.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building a house, but instead of laying bricks yourself, you ask a super-fast, incredibly talented robot to do it. You give the robot a simple sentence like, "Build me a cozy cottage with a red door," and within seconds, it hands you a perfect-looking home. This is the world of "vibe coding," where artificial intelligence (AI) writes entire working apps from short prompts. It's like magic: you get a finished product instantly. But here's the catch: the robot is great at making things look good and work fast, but it often forgets the boring, invisible safety stuff. It might leave the front door unlocked, forget to install smoke detectors, or leave the back gate wide open because those things don't make the house look pretty in a quick tour.
In the world of computer security, these invisible gaps are called vulnerabilities. Think of them as hidden cracks in the foundation or windows that don't lock. For a professional team, there are security guards (human experts) who check these things before anyone moves in. But for the solo builder using AI, there's no one else checking. The robot just hands over the keys, and the builder might not even know the house is unsafe. This paper asks a simple, crucial question: If we build the exact same house twice—one time with the robot alone, and another time with the robot plus a special "security inspector" robot that checks its own work—how much safer does the second house become? The researchers wanted to see if adding a simple, automated check could turn a shaky, unsafe app into a fortress without slowing down the fun, fast vibe of building it.
The Experiment: Two Apps, One Big Difference
The researchers set up a controlled experiment using a fake online store called "Northwind." They asked an AI to build this store twice. The first version, let's call it App A, was built with the robot alone. The second version, App B, was built with the same robot, but this time, the robot had a special "security audit skill" attached to it. This skill is like a tireless inspector that runs a five-step loop: it scans the code, flags problems, fixes them, and then scans again to make sure the fix actually worked.
The key to this experiment was that the only difference between the two apps was this security skill. The prompt given to the AI was identical, and the core features (the products, the shopping cart, the checkout) were exactly the same. This ensured that any difference in safety wasn't because the AI got lucky or changed the design, but purely because of the security inspector.
The Results: A Massive Safety Boost
The results were dramatic. When they scored both apps on a scale of 100 based on 16 different security rules (like having a "Content Security Policy," which is a rulebook telling the browser which scripts it's allowed to run, and a "Referrer Policy," which stops the app from leaking your browsing history to other sites), the difference was huge.
- App A (No Security Skill): It scored a 58 out of 100. It had 16 security problems, including four "high-severity" gaps. These were the dangerous holes, like missing locks on the front door. The total "risk" was measured at 100 units.
- App B (With Security Skill): It scored a 99 out of 100. The security skill fixed almost everything. It eliminated all four high-severity gaps and reduced the total number of problems from 16 down to just 1. The total risk dropped from 100 units to 1 unit.
In plain numbers, the security skill cut the unmitigated risk by 99.0%. The single remaining risk was a specific check that requires a real backend server (which this demo didn't have), so the skill honestly flagged it as something that still needed human attention rather than pretending it was fixed.
Why This Matters: The "Glitch" Bonus
One of the most interesting findings wasn't just about stopping hackers, but about preventing the app from breaking itself. The researchers found that if you just add strict security rules to a messy app, the app often breaks. For example, if you tell a browser "No outside scripts allowed," but the app has code that tries to use outside scripts, the app stops working.
The security skill was smart enough to realize this. Before it added the strict rules, it went back and cleaned up the code, moving styles and scripts into the right places so the app would still look and work perfectly. It prevented "glitches" that a human builder might not notice until it was too late. It didn't just patch holes; it reinforced the whole structure so the safety rules wouldn't cause the house to collapse.
What the Paper Says (and Doesn't Say)
The paper is very clear about what it doesn't prove. It does not claim that the app is now "invincible" or that you can stop worrying about security entirely. The authors emphasize that this is a static demo (a fake store with fake payments) and that real-world apps with real money and real user data still need professional human security experts to do deep penetration testing. The 99% improvement is specific to the controls the skill can automate in this type of app.
However, the paper strongly suggests that for the millions of small apps being built by hobbyists and solo developers using AI, this kind of automated, built-in audit is a game-changer. It suggests that by baking security checks directly into the AI's workflow, we can raise the "safety floor" of software by a massive amount, turning what used to be a dangerous habit of "vibe coding" into something much more reliable. The study concludes that this approach is a cheap, high-impact step that makes AI-generated software roughly two orders of magnitude safer, simply by making the safe path the default path.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.