Program Analysis Guided LLM Agent for Proof-of-Concept Generation
The paper introduces PAGENT, a novel hybrid framework that combines program analysis (both static and dynamic) with an LLM agent to significantly improve the scalability and success rate of automated Proof-of-Concept generation for software vulnerabilities, outperforming prior top-performing approaches by 132%.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Problem: The "Missing Key"
Imagine you are a security guard (a software developer) at a massive, complex castle (a software project). Someone sends you a note saying, "Hey, there's a loose brick in the wall behind the kitchen that lets intruders in!"
This is a vulnerability report. But here's the catch: the note doesn't tell you exactly how to get to that brick, or what kind of rock you need to throw to knock it loose.
To fix the wall, you first need to prove the hole exists. You need to build a Proof-of-Concept (PoC)—a specific, perfect rock throw that hits the loose brick and makes it fall.
The Problem:
- Old Tools (Fuzzing/Symbolic Execution): These are like blindfolded monkeys throwing rocks. They throw millions of rocks randomly. Sometimes they hit, but often they get stuck in the castle's maze (the code) and never reach the kitchen. They need a human expert to guide them, which is slow and expensive.
- New Tools (AI/LLMs): These are like brilliant detectives who can read the castle blueprints. But, they have a bad habit of hallucinating. They might confidently say, "I know exactly where the brick is!" and throw a rock at a solid wall, thinking it's the loose one. They are smart but lack precision.
The Solution: PAGENT (The "Super Detective" with a Flashlight)
The authors created PAGENT, a system that combines the best of both worlds. It's like giving the brilliant detective a flashlight (Static Analysis) and a drone (Dynamic Analysis) to guide them.
Here is how PAGENT works, step-by-step:
1. The Flashlight: Static Analysis (The Map Maker)
Before the detective (the AI) starts guessing, PAGENT uses a "Flashlight" to scan the castle blueprints.
- What it does: It doesn't try to solve the whole puzzle. Instead, it draws a simple map showing only the paths that lead to the kitchen. It filters out all the dead ends and irrelevant rooms.
- The Analogy: Imagine the AI is trying to find a needle in a haystack. The Static Analysis doesn't find the needle, but it burns away 99% of the hay so the AI only has to look at a small, manageable pile. It tells the AI: "The loose brick is in the kitchen, and here is the exact hallway you must walk down to get there."
2. The Detective: The LLM Agent (The Problem Solver)
Now, the AI gets the map. It uses its "brain" (Large Language Model) to figure out the specific details: What size rock do I need? Do I need to open a specific door first?
- The Analogy: The AI is the detective who writes the plan. It looks at the map and says, "Okay, I need to turn left at the fountain, then climb the ladder." It generates a candidate PoC (a test rock throw).
3. The Drone: Dynamic Analysis (The Reality Check)
The AI throws its rock. But did it hit the brick?
- What it does: PAGENT runs the rock throw in a safe, simulated environment (a test kitchen). It watches exactly what happens.
- If it hits: Great! We found the vulnerability.
- If it misses: The "Drone" sends a video back to the AI. It says, "You threw the rock, but you hit the door instead of the kitchen. Also, you didn't even get past the first hallway."
- The Analogy: This is the feedback loop. The AI doesn't just guess and give up. It gets a report card: "You missed the target by 5 feet. Try aiming higher." The AI then rewrites its plan and tries again.
Why is this a Big Deal?
The paper tested PAGENT on 203 real-world software vulnerabilities (like holes in famous open-source tools).
The Results:
- Standard AI agents (without the flashlight or drone) were like students guessing on a test; they got about 18% of the answers right.
- PAGENT, using a cheaper, open-source AI model, got 64.6% right.
- The Magic: PAGENT didn't just do better; it did 132% better than the previous best methods. It found vulnerabilities that the "smartest" expensive AI models missed completely.
The "Post-Patch" Surprise:
Sometimes, developers fix a hole, but they miss a tiny crack nearby. PAGENT was so good at exploring the castle that it found 32 new holes in the "fixed" versions of the software that the original developers didn't even know were there!
The Bottom Line
Think of PAGENT as a Cyber-Construction Crew:
- The Architect (Static Analysis) draws the precise blueprints so no time is wasted.
- The Foreman (AI Agent) uses those blueprints to figure out the best way to break the wall.
- The Inspector (Dynamic Analysis) watches the break, tells the Foreman what went wrong, and sends them back to try again until the wall actually falls.
By combining human-like reasoning (AI) with strict, mathematical rules (Program Analysis), PAGENT automates the hardest part of software security: proving that a bug is real so it can be fixed. It turns a job that used to take days of expert work into a process that takes minutes, saving time and money while making software safer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.