SecureVibeBench: Evaluating Secure Coding Capabilities of Code Agents with Realistic Vulnerability Scenarios

1. Problem Statement

Large Language Model (LLM) powered code agents are transforming software engineering, but their ability to generate secure code remains a critical concern. Existing benchmarks for evaluating secure coding suffer from three major limitations that prevent fair comparisons between human developers and AI agents:

• Unrealistic Task Form: Most benchmarks focus on single-function completion or "fill-in-the-middle" tasks, failing to capture the complexity of real-world software maintenance which involves multi-file edits across large repositories.
• Misaligned Context: Many benchmarks synthesize tasks from vulnerability catalogs (e.g., CWE) or use simplified contexts. They often fail to reconstruct the exact scenario where a human developer originally introduced a vulnerability, making it impossible to determine if agents replicate human errors or introduce novel risks.
• Incomplete Evaluation: Prior works often neglect functional correctness (a prerequisite for security) or fail to detect new security risks introduced by agents that were not present in the original vulnerability.

2. Methodology: SECUREVIBEBENCH

The authors introduce SECUREVIBEBENCH, a benchmark designed to evaluate code agents in realistic, repository-level secure coding scenarios.

A. Data Collection & Construction

• Source: The benchmark is built upon OSS-Fuzz and ARVO (Atlas of Reproducible Vulnerabilities), covering 41 critical C/C++ open-source projects (e.g., OpenSSL, harfbuzz).
• Vulnerability Backtracking: To reconstruct the exact scenario where a human introduced a vulnerability, the authors propose a cascading static and dynamic analysis approach:
  1. Static Analysis: Uses the B-SZZ algorithm to identify potential "Vulnerability-Introducing Commits" (VIC) candidates efficiently.
  2. Dynamic Analysis: Verifies candidates using Proof-of-Vulnerability (PoV) programs. A candidate is valid only if the parent commit is secure, the candidate commit is vulnerable, and the fixing commit is secure.
• Task Formulation:
  • Input: A natural language requirement (synthesized from commit messages and issue trackers, stripped of security-specific details) and the repository state before the vulnerability was introduced.
  • Output: A code patch fixing the requirement.
  • Scope: Requires editing multiple files across the repository (average 1.9 files, up to 5 files; average 42.5 lines of code).

B. Evaluation Framework

The benchmark employs a comprehensive, multi-dimensional evaluation oracle:

1. Functionality: Uses differential testing against the repository's test suite to ensure the agent's patch behaves identically to the "gold" (human) patch.
2. Security (Specific): Uses PoV programs to check if the specific historical vulnerability is reintroduced.
3. Security (General/New): Uses Static Application Security Testing (SAST) (Semgrep) to detect new security risks introduced by the agent that were not present in the original code.
4. Categorization: Solutions are classified into four types:
   • Incorrect (IC): Fails functional tests.
   • Correct but Vulnerable (C-VUL): Passes tests but contains the historical vulnerability.
   • Correct but Suspicious (C-SUS): Passes tests, no historical vulnerability, but SAST detects new risks.
   • Correct and Secure (C-SEC): Passes all functional and security checks.

3. Key Contributions

1. First Repository-Level Secure Coding Benchmark: SECUREVIBEBENCH is the first benchmark to evaluate secure code generation at the repository level with multi-file editing capabilities, moving beyond function-level completion.
2. Aligned Vulnerability Context: It introduces a novel methodology to precisely backtrack and reconstruct the exact code version and requirements where humans introduced vulnerabilities, enabling a fair "human vs. agent" comparison.
3. Holistic Evaluation: It combines functional testing, specific vulnerability checks, and SAST to detect both historical and novel security risks, providing a more realistic assessment of agent safety.

4. Experimental Results

The authors evaluated 5 code agents (SWE-agent, OpenHands, Aider, Claude Code, Codex) powered by 5 LLMs (including Claude Sonnet 4.5, GPT-5, DeepSeek-V3.1) on 105 curated tasks.

• Overall Performance: Current agents struggle significantly. The best-performing combination (SWE-agent + Claude Sonnet 4.5) achieved a C-SEC rate of only 23.8%.
• Functionality vs. Security:
  • While some agents achieve high functional correctness (e.g., SWE-C4.5 at ~46.7% correct), a large portion of these correct solutions still contain vulnerabilities or new security risks.
  • For example, in the OH-C4.5 configuration, over 50% of the correct solutions were either vulnerable or suspicious.
• Failure Modes:
  • Agents: SWE-agent and OpenHands showed similar failure patterns (roughly 40% compilation errors, 40% test failures). Aider frequently failed due to context window limits (loading too many files at once).
  • Models: Claude Sonnet 4.5 outperformed other models significantly in secure coding tasks.
• New Risks: Agents introduced 292 "suspicious" cases covering 16 CWEs, with 71.6% related to memory safety (e.g., Use-After-Free, Buffer Overflow). Notably, agents introduced vulnerability types (e.g., CWE-14, CWE-319) not present in the original dataset, indicating they generate novel risks.
• CLI Agents: Customized CLI agents like Claude Code and Codex showed high functional correctness but lagged in security, reinforcing that current scaffolds prioritize functionality over safety.

5. Significance and Implications

• Security Gap: The results highlight a critical gap: even state-of-the-art agents cannot reliably generate code that is both functionally correct and secure in realistic, complex environments.
• Novel Threats: The finding that agents introduce new types of vulnerabilities (beyond the historical ones) suggests that simply training on secure code or fixing known CVEs is insufficient; agents require specific security alignment and reasoning capabilities.
• Benchmark Standard: SECUREVIBEBENCH sets a new standard for evaluating AI in software engineering by demanding repository-level context, precise vulnerability backtracking, and multi-faceted security validation.
• Future Directions: The paper calls for improved long-context handling, better security-aware reasoning in LLMs, and the development of more robust evaluation frameworks that account for the unique failure modes of AI-generated code.