CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training

Technical Summary: CyberForge

Problem Statement
Frontier Large Language Model (LLM) agents currently struggle to discover and patch complex vulnerabilities in real-world software. While these agents can assist attackers who only need to find a single exploitable weakness, defenders must continuously secure entire codebases. The primary bottleneck in developing stronger defensive agents is the scarcity of high-quality security training data that includes reproducible build and execution environments. Existing datasets are largely limited to historical Common Vulnerabilities and Exposures (CVEs) and bug-bounty reports, which are assembled manually or via automated replay of disclosed issues. These resources are constrained by the rate of human discovery and public disclosure, and they often fail to scale as training data because validating a vulnerability requires a dual condition: the injected code must pass all existing unit tests (remaining latent) while being triggerable by a specific adversarial input (Proof of Vulnerability, or PoV).

Methodology: The CyberForge Framework
CyberForge is a framework designed to synthesize executable, repository-level security training data by injecting vulnerabilities into real C/C++ projects. It decouples corpus growth from the rate of human disclosure by creating new instances rather than mining historical ones. The framework operates on projects enrolled in OSS-Fuzz, leveraging their pre-existing Docker images, build scripts, and libFuzzer harnesses to ensure reproducible environments.

CyberForge employs two complementary injection pipelines:

1. Fuzzer-Guided Injection: This pipeline utilizes existing fuzzing infrastructure to identify high-value injection sites. It parses OSS-Fuzz metadata to map functions reachable by existing harnesses. A scoring system ranks these functions based on structural role, call depth, and "triggerability" (likelihood of being reached by an existing harness). An LLM agent then performs a minimal, targeted modification to weaken a security check at the selected site. Validation is automated: the modified build must pass all unit tests, and a PoV is generated to trigger the vulnerability. A post-hoc libFuzzer run with a format-aware seed corpus provides an additional validation path.
2. Agentic In-Context Injection: This pipeline targets vulnerabilities beyond the reach of existing fuzz harnesses. It uses a hybrid retrieval strategy, combining structural (AST-based) and semantic (embedding-based) similarity to match target projects with historical CVE patterns from the PrimeVul dataset. Additionally, specialist LLM agents explore the codebase directly to propose diverse weakness types. The injection process involves iterative retry loops, taint analysis to verify dataflow paths from input sources to the vulnerable sink, and PoV generation using sanitizer reports (ASAN/UBSAN) and observable side effects.

Validation Mechanism
A shared Differential PoV Oracle validates all generated instances. An instance is accepted only if:

• Condition 1: The injected build passes all existing unit tests of the project (ensuring the vulnerability is latent and not a functional bug).
• Condition 2: The generated PoV triggers a failure (e.g., sanitizer error, crash) on the injected build but not on the clean build under identical input.

Key Contributions

• Repository-Level Generation Framework: CyberForge synthesizes training data from buildable codebases using vulnerability injection and differential validation, moving beyond function-level static injection.
• Validated Corpus: The framework produced 1,034 validated vulnerabilities across 80 C/C++ projects and 63 weakness categories (CWEs). The edit locality of these synthetic patches closely resembles real CVE patches, with a Kolmogorov–Smirnov distance of 0.165 compared to a 0.190 baseline between two real CVE corpora.
• Downstream Performance Improvements: Fine-tuning agents on trajectories collected from this corpus significantly improved performance on the SEC-bench (in-domain) and PatchEval (out-of-domain) benchmarks.

Results
The authors fine-tuned three open-source student models (Gemma 4 E4B, 12B, and 31B) using trajectories collected from two teachers (Gemma 4 31B and GPT-5.4-mini).

• SEC-bench Performance: All six student-teacher configurations improved patch repair scores by +3.3 to +14.7 points. The strongest configuration (31B student with GPT-5.4-mini teacher) achieved 72.7%, approaching the teacher's 74.0%.
• Out-of-Distribution Generalization: The gains transferred to PatchEval, a benchmark containing Go, JavaScript, and Python tasks, despite the training corpus being exclusively C/C++. Every configuration improved on this cross-language benchmark.
• Behavioral Analysis: Fine-tuning steered agents toward their teachers' workflows. Smaller models (e.g., 12B) gained primarily by learning to reach the verification stage (increasing workflow coverage), while larger models (31B) improved their success rate once the verification stage was reached.
• Scaling: Performance scaled monotonically with the number of training trajectories, showing no signs of saturation at 880 trajectories.

Significance and Claims
The paper claims that CyberForge provides a scalable source of execution-validated data for training cybersecurity agents, addressing the critical training-data gap on the defensive side of AI-assisted security. By generating synthetic vulnerabilities that are verified through execution rather than static analysis, the framework produces realistic training instances that are not limited by the pace of public disclosure. The authors emphasize that their framework generates proofs of vulnerability (PoVs) to demonstrate the existence and reachability of weaknesses, rather than producing active exploits designed to compromise systems. The work demonstrates that targeted training on synthesized, repository-level data can significantly enhance the ability of AI agents to locate and patch security flaws in complex software environments.