Technical Summary: From Runnable to Verifiable

Problem Statement

The security research community is increasingly relying on Large Language Models (LLMs) and agentic systems to automate vulnerability validation, including the reproduction of known CVEs, generation of Proof-of-Concept (PoC) exploits, and code repair. While a growing body of literature claims success in these tasks, the quality of the accompanying artifacts varies significantly.

Current reproducibility studies typically measure "availability" (is the code public?) and "runnability" (does it build?). However, there is a critical, unmeasured gap between an artifact that merely runs and produces a signal (e.g., a crash or a log message) and one that provides semantic confirmation that the specific CVE was actually reproduced. This study addresses the lack of operationalized metrics to distinguish between artifacts that simply "run and print something alarming" and those that rigorously validate a specific vulnerability claim against ground truth.

Methodology

The author conducted a pre-registered, independent reproducibility audit of LLM/agent-driven vulnerability validation literature. The study employs a four-level evidence ladder to grade artifacts:

1. Available: The artifact is publicly reachable.
2. Runnable: The artifact completes its workflow on a clean snapshot (R0) or after minimal environment-compatible repair (R1), without modifying the exploit logic or oracle.
3. Signal-Producing: Execution yields candidate signals (crashes, status changes, or specific markers).
4. Semantically Confirmed: The signal matches a CVE-specific post-condition, validated via:
   • G3a (Negative Control): The signal is absent on benign input.
   • G3b (Patched Counterfactual): The signal is absent on the patched build.

Corpus and Sampling:

• Search: A frozen search protocol (2023–2026) across arXiv, DBLP, and OpenAlex yielded 104 papers in a consensus corpus.
• Availability: 59 of 104 papers (56.7%) had publicly reachable artifacts.
• Paper-Level Sample: 18 papers were selected to measure R0/R1 runnability and repair burdens.
• Case-Level Sample: The study focused heavily on an "anchor benchmark" (arXiv:2509.24037) containing 102 distinct vulnerability cases. A subset of 20 cases served as a calibration pilot, while the remaining 82 were executed to form a full-corpus analysis.

Execution Protocol:

• R0: Execution on a clean, frozen snapshot with zero modification.
• R1: Execution after environment-only repairs (e.g., installing missing dependencies, fixing EOL apt sources) without touching exploit logic, oracles, or post-conditions.
• Oracle Audit: The study independently verified the reliability of the "artifact-embedded oracles" (the scripts' internal logic for declaring a success) against the patched counterfactual ground truth.

Key Findings

1. The Runnability Gap

Among the 18 paper-level artifacts tested:

• Only 10/18 (55.6%) completed their declared workflow on a clean snapshot (R0).
• After environment-only repairs (R1), this rose slightly to 11/18 (61.1%).
• The remaining 7 artifacts failed at both levels. Dominant failure modes included missing dependency manifests, incompatible package managers (e.g., uv vs. pip), and reliance on external services or datasets not included in the artifact.

2. Script-Internal Divergence

A significant structural issue was identified in the anchor corpus: 58/102 (56.9%) cases contained a script-internal CVE identifier that diverged from the directory's declared CVE.

• Example: A directory declared CVE-2023-25668 but the script executed a trigger for CVE-2023-25676.
• This divergence undermines the validity of the reproduction claim before execution even begins and is invisible to standard availability audits.

3. Unreliability of Artifact-Embedded Oracles

The study found that the "oracles" (the internal logic determining if a PoC succeeded) are systematically unreliable.

• Patched Counterfactuals (G3b): Of 30 cases where a signal was produced and a patched build was audited, 20/30 (66.7%) still produced the claimed signal on the patched build. These "dirty" signals indicate the PoC triggers a generic crash or behavior unrelated to the specific CVE fix.
• Negative Controls (G3a): Of 19 cases tested against benign input, 7/19 triggered false positives.
• Oracle Performance: The confusion matrix for artifact-embedded oracles showed a sensitivity of 60% and specificity of 45%.
  • Two-thirds of oracle triggers were false positives (signals that persist on patched builds).
  • Genuine CVE-specific signals were missed by the oracle in 4/10 cases (often because the oracle looked for a specific marker file that wasn't created, even though a real exploit occurred).

4. The Semantic Confirmation Gap

Strict confirmation (E1), requiring a signal that passes G2 (post-condition match), G3a (negative control), and G3b (patched counterfactual), was achieved in only 2 cases out of the signal-producing corpus.

• The study highlights that "availability → runnable → signal-producing" does not imply "semantically confirmed."
• A trigger on a vulnerable build is not evidence of CVE-specific reproduction without a clean patched counterfactual.

Significance and Contributions

The paper positions itself as an exploratory, pre-registered audit that operationalizes the "semantic confirmation gap" for the first time in this domain. Its primary contributions are:

1. A Reproducibility Protocol: A reusable template for the security community featuring:
   • CVE-specific post-conditions.
   • An R0/R1 repair ladder.
   • G1–G3 semantic evidence levels.
   • Patched-counterfactual oracles to distinguish specific CVE reproduction from generic crashes.
2. Empirical Evidence of Failure: The study demonstrates that the majority of "successful" LLM/agent validation artifacts fail to provide semantic confirmation. Specifically, most signal-producing cases do not survive their own patched counterfactuals, and internal oracles are poor proxies for ground truth.
3. Open Evidence Bundles: The author releases frozen protocols, screening decisions, claim sheets, and per-case execution logs to serve as a baseline for future reproducibility studies.

The author concludes that the security community must move beyond trigger-string checks and require patched-counterfactual verification to validate claims of LLM/agent-driven vulnerability reproduction. The study notes that while these are exploratory results from a single anchor corpus and a small paper sample, the identified failure modes (divergent CVE IDs, unreliable oracles, lack of patched verification) suggest a systemic issue in current artifact quality.