SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories

1. Problem Statement

While Large Language Model (LLM) powered code agents are increasingly used to automate software development, they frequently generate code containing security vulnerabilities. Existing benchmarks for evaluating LLM security suffer from two critical limitations when applied to real-world scenarios:

1. Lack of Dual Evaluation: Most benchmarks evaluate either functional correctness or security, but rarely both simultaneously on the same dataset. Evaluating security without correctness can lead to overestimating safety (e.g., code that is secure but non-functional).
2. Insufficient Context: Existing repository-level benchmarks often provide fixed context retrieval methods. They fail to evaluate agents that utilize dynamic, customized context retrieval (e.g., navigating file structures, looking up definitions) across an entire codebase.

There is a need for a benchmark that simulates realistic Integrated Development Environment (IDE) scenarios where agents must complete code within a specific region of a real-world repository, ensuring both functional correctness and security.

2. Methodology: SecRepoBench

The authors introduce SecRepoBench, a repository-level benchmark designed to evaluate code agents on secure code completion tasks.

• Dataset Construction:

  • Source: Built upon the ARVO dataset, which contains over 5,000 reproducible security vulnerabilities in C/C++ projects identified by Google OSS-Fuzz.
  • Selection Criteria: The authors filtered for tasks where the vulnerability fix modified only one function within a single file. This ensures the task is a focused code completion problem while still requiring repository-level context.
  • Scale: The final benchmark consists of 318 code completion tasks across 27 popular GitHub C/C++ repositories, covering 15 Common Weakness Enumerations (CWEs).
  • Task Format: Each task presents a target function with a "masked" region (the vulnerable code removed) and the surrounding repository context. The agent must generate the code to fill this region.

• Evaluation Metrics:

  • Functional Correctness: Evaluated using developer-written unit tests from the original repositories. A completion is correct only if it passes all relevant unit tests that the ground-truth secure patch passes.
  • Security: Evaluated using OSS-Fuzz Proof-of-Concept (PoC) exploits. If the generated code causes the program to crash when executed with the PoC input, it is deemed vulnerable.
  • Primary Metric: Secure-pass@1, which measures the percentage of tasks where the generated code passes both the unit tests and the security test.

• Benchmark Features:

  • Context Retrieval: Unlike static benchmarks, SecRepoBench allows agents to explore the entire repository.
  • Neutrality: Prompts are security-neutral (no explicit warnings about vulnerabilities), forcing agents to identify risks based on context alone.
  • Mutation: To prevent memorization, variable names in the masked regions are semantically mutated using GPT-4o before evaluation.

3. Key Contributions

1. Novel Benchmark: SecRepoBench is the first repository-level benchmark that simultaneously evaluates functional correctness and security using real-world developer unit tests and dynamic fuzzing inputs.
2. Agent Framework Evaluation: It evaluates not just standalone LLMs but also 15 code agents built on three state-of-the-art frameworks (OpenHands, Aider, Claude Code) combined with various LLMs.
3. Comprehensive Analysis: The paper provides a deep dive into failure modes, comparing the performance of reasoning vs. non-reasoning models, and analyzing the trade-offs between correctness and security in agent-generated code.

4. Experimental Results

The authors evaluated 29 standalone LLMs and 15 code agent configurations on SecRepoBench.

• Standalone LLMs Struggle: Even the best-performing standalone model (GPT-5) achieved only 39.3% Secure-pass@1. Most models struggle to generate code that is both correct and secure.

• Agents Outperform Standalone Models: Code agents significantly outperformed standalone LLMs. The best setup (OpenHands + OpenAI o3) achieved 53.5% Secure-pass@1.

• Difficulty Comparison: SecRepoBench is significantly harder than the previous state-of-the-art benchmark, BaxBench.

  • GPT-5 scored 53.8% on BaxBench but dropped to 39.3% on SecRepoBench.
  • The best agent setup (53.5%) did not even surpass the best standalone model's performance on BaxBench (53.8%), highlighting the increased difficulty of real-world repository contexts.

• Correctness vs. Security Trade-off:

  • The performance gain from using agents is primarily driven by improved functional correctness (pass@1 increased by ~18.3% on average).
  • The improvement in security (secure %) was much smaller (~14.5% on average).
  • Agents often prioritize fixing logic errors over identifying security vulnerabilities, leading to code that works but remains insecure.

• Failure Mode Analysis:

  • Compilation Errors: Agents frequently hallucinate non-existent identifiers, struct members, or macros (8.5% of tasks), indicating poor understanding of repository-wide dependencies.
  • Missing/Incorrect Checks: In "Correct but Insecure" cases, agents often fail to add necessary boundary checks or implement them incorrectly, despite having access to the relevant logic.

5. Significance

• Realistic Assessment: SecRepoBench provides a more rigorous and realistic evaluation of code agents than previous benchmarks, reflecting the complexities of real-world software development (dependencies, build systems, and existing codebases).
• Identifying Gaps: The results reveal a critical gap in current AI coding tools: while agents are becoming better at making code work, they are not yet sufficiently reliable at making code safe.
• Future Directions: The paper suggests that future research must focus on improving agents' ability to:
  1. Maintain consistency with existing codebases to avoid hallucinations.
  2. Explicitly reason about security implications during the code refinement process, rather than just functional logic.
  3. Integrate security checks as a core part of the agent's iterative refinement loop.

In conclusion, SecRepoBench establishes a new standard for evaluating secure code generation, demonstrating that while code agents offer significant improvements over standalone LLMs, substantial challenges remain in ensuring the security of AI-generated code in real-world repositories.