💻 computer science

One Successful Run Is Not an Exploit: Replaying LLM-Written Exploit Scripts with the Model Out of the Loop

This paper demonstrates that single-run success in LLM-based offensive security benchmarks is often misleading by showing that while generated exploit scripts are highly stable when replayed without the model, the actual instability lies in the generation process rather than the scripts themselves, necessitating a separation of model performance from artifact reliability for accurate evaluation.

Original author: Wensheng Huang

Published 2026-11-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original author: Wensheng Huang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer security, there is a constant race between those who build defenses and those who try to break them. When a computer system has a hidden weakness, known as a vulnerability, a skilled attacker can write a small piece of code to exploit it and gain unauthorized access. Today, researchers are using artificial intelligence to write these pieces of code automatically. These AI systems, often called agents, are given a challenge and a set of rules, and they try to generate a script that successfully breaks into a test system. If the script works even once, the standard way of measuring success in this field is to count it as a victory. The assumption has been that if an AI can write a script that works once, it has found a reliable way to break the system. But this assumption leaves a critical question unanswered: if you take that exact same script and run it again on a fresh system, will it work a second time?

A researcher set out to answer this question by changing how they tested these AI-generated scripts. Instead of asking the AI to try again and again, which would test the AI's ability to think and adapt, they decided to test the scripts themselves. They took the winning scripts that the AI had already produced and ran them repeatedly on brand-new, clean copies of the target systems, without letting the AI participate in the second round. This approach is like taking a key that successfully opened a door once and trying it on a dozen new, identical doors to see if it still fits. The researcher wanted to know if the key was truly good, or if it only worked because of a specific, temporary condition of the first door.

The study involved fourteen different security challenges, ranging from simple login flaws to more complex issues where the system processes user input in dangerous ways. The AI was asked to solve these challenges, and it succeeded in generating working scripts for 196 different attempts. The researcher then took each of these 196 scripts and ran them ten times each on fresh systems, creating a total of 1,960 test runs. The results were strikingly clear: in 1,950 of those runs, the script worked exactly as it had the first time. This means that for the vast majority of cases, once the AI found a solution, that solution was a stable, reliable piece of code that could be reused. The script did not rely on luck or a fleeting moment of success; it was a solid tool that worked consistently.

However, the study also uncovered a fascinating exception that explains why the old way of testing was insufficient. There was one script out of the 196 that failed every single time it was run on a fresh system. It was a "dead" script that could not open the door. Yet, when the researcher ran the entire sequence of steps the AI had taken to create that script—replaying the AI's previous attempts in order before running the final script—the script suddenly worked perfectly every time. The reason was subtle and hidden. The script relied on the internal state of the computer system, specifically a list of classes that the system's programming language kept in memory. The AI's earlier attempts had unknowingly added items to this list, shifting the positions of the items. The final script was hard-coded to look for specific items at specific positions. On a fresh system, those positions were empty or held the wrong things. But after the earlier steps ran, the list had grown and shifted, placing the correct items exactly where the script expected them. The success belonged to the entire journey, not just the final step.

This discovery highlights a major difference between how the AI thinks and how the code it produces behaves. The researcher found that the AI's ability to generate a solution was quite unstable. In many cases, the AI would fail to find a solution on its first try, succeed on the second, and then fail again on the third. If you only looked at the AI's generation process, you would see a lot of inconsistency. But once the AI managed to write a script that worked, that script itself was almost always stable. The instability was in the thinking process, not in the tool that was created. This means that a single successful run by an AI does not prove that the AI is consistently good at finding solutions, but it does prove that the specific solution it found is usually reliable.

The researcher also tested whether these scripts would work if the computer system they were attacking was slightly different, such as running a newer version of the programming language. They rebuilt the target systems with a slightly updated version of the software and ran the scripts again. Almost all of them still worked, but one script that had been perfectly reliable on the old system failed completely on the new one. This showed that even a small change in the environment could break a script that seemed perfect, reinforcing the idea that a script's success is tied to the specific conditions under which it was created.

The main takeaway from this work is that we need to change how we judge the success of AI in security. A single success is not enough to call a script an exploit. To truly know if an AI has found a real vulnerability, we must verify that the script it wrote works repeatedly on its own, without the AI's help. The study shows that while the AI's generation process is often shaky and unpredictable, the scripts it manages to produce are usually solid and dependable. The one time a script failed to work on its own, it was because it depended on a hidden state left behind by the AI's earlier attempts, a detail that a single test run would never reveal. By separating the act of writing the code from the act of running it, the researcher provided a clearer picture of what these AI systems are actually capable of, showing that the tools they build are often more reliable than the process that builds them.

Technical Summary: One Successful Run Is Not an Exploit

Problem Statement
Current offensive-security benchmarks for Large Language Model (LLM) agents typically define a "successful" run as a single execution where the agent's output contains a secret flag string. This convention conflates the success of the generation process (the model producing a working script) with the stability of the artifact (the script itself). A single verified run does not answer whether the model will produce the same result again, nor does it verify if the generated script works independently of the specific target state accumulated during the agent's interaction. Existing literature often stops at the first success or repeats the agent generation rather than the artifact, failing to isolate the script's reliability from the model's sampling variance or the target's accumulated state.

Methodology
The author proposes a protocol that separates the generation layer from the artifact layer by taking the model "out of the loop." The study utilizes a corpus of 14 web challenges (10 authored, 4 re-implementations of CTF challenges) across 28 interface variants.

Generation Layer: An LLM agent (Qwen3.8-flash) attempts to solve the challenges with a fixed prompt budget (up to 5 rounds, 300s time limit). 280 runs were executed (10 runs per cell).
Artifact Selection: From the 196 successful runs, the source code of the first round that produced the flag was extracted as the "winning artifact."
Replay Arms (Model Out of the Loop):
- Cold Replay: Each of the 196 winning scripts was executed verbatim $R=10$ times against a freshly started target container. No model calls were made; the script text was submitted directly to the driver.
- Sequence Replay: For artifacts derived from multi-round runs, the entire ordered sequence of executed scripts from that specific run was replayed to isolate the effect of accumulated target state.
- Environment Replay: The 195 stable artifacts were replayed against a rebuilt execution image (Python 3.13.15 to 3.13.18) to test environmental sensitivity.
Metrics: Success is binary (flag present in output). Artifacts are classified as Stable (10/10 hits), Flaky (1–9 hits), or Dead (0/10 hits). Statistical analysis uses Clopper-Pearson intervals and task-clustered bootstrapping.

Key Contributions

C1 (Artifact-Level Replay Protocol): The study defines and executes "cold replay" of verified scripts, reporting a replay rate of 99.49% (1,950/1,960) with a 95% confidence interval of 98.37% to 100.00%. It classifies 195 artifacts as stable, 0 as flaky, and 1 as dead under the reference environment.
C2 (Separation of Instability Sources): The paper demonstrates that instability resides primarily in the generation layer, not the artifact layer. While 50% of the generation cells (12 of 24 with hits) were "flaky" (producing hits in fewer than 10/10 runs), 11 of those 12 cells contained artifacts that were perfectly stable (10/10) upon cold replay. The single "dead" artifact failed because it hard-coded indices dependent on a target process state primed by earlier rounds in its specific trajectory, not because the script logic was inherently broken.
C3 (Environment Binding): A patch-level rebuild of the execution environment (Python version update) caused one additional artifact (a constrained SQL injection script) to flip from stable (10/10) to dead (0/10), highlighting that even stable artifacts are bound to specific environment definitions.

Results

Replay Stability: The vast majority of LLM-written exploits (195/196) are reproducible artifacts. When a script works once, it almost always works again when replayed in isolation.
The Exception (State Dependence): The single dead artifact (t03) failed 0/10 times in cold replay but succeeded 10/10 times when the full trajectory (including non-hitting preliminary requests) was replayed. This script relied on a subclass table in the target interpreter that was inflated by prior requests. A single "priming" request moved the script from 0/10 to 10/10 success, proving the hit belonged to the trajectory, not the script alone.
Generation vs. Artifact: There is a systematic disagreement between the two layers. A model may be "flaky" in generating a solution (e.g., succeeding only 8/10 times), but the specific script it generates when it does succeed is often stable. Conversely, a single-run score cannot distinguish between a model that reliably writes stable scripts and one that occasionally writes a script that only works within its own specific interaction history.

Significance and Claims
The paper claims that a single-run score reports a property of the generator (the model's ability to produce a hit on a specific attempt) rather than the artifact (the reusability of the code).

Measurement Validity: Re-running an agent measures generation variance; replaying an artifact measures script stability. These are distinct quantities.
Protocol Recommendation: The author proposes that benchmarks should report both the generation layer success rate (with $n$ trials stated) and the artifact layer replay rate (with the model out of the loop).
Modest Scope: The author explicitly states this is not a general claim about all LLM exploits, nor a model comparison. The findings are specific to the corpus (14 challenges, one model, one decoding config). The "stable" classification is relative to the specific environment and trajectory; it does not imply operational reliability against diverse real-world targets.
Practical Implication: The field's current practice of stopping at the first success obscures the fact that half of the "successful" configurations in this study were actually flaky at the generation level, while the scripts themselves were largely stable. The title "One Successful Run Is Not an Exploit" is supported by the finding that the score fails to transfer, even when the script does, and in the one case where the script fails to transfer, the reason is unrecorded state.

Technical Summary: One Successful Run Is Not an Exploit

More like this