Technical Summary: SEC-bench Pro

Problem Statement

Large Language Models (LLMs) are increasingly deployed for automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. However, existing benchmarks fail to faithfully evaluate LLMs in real-world bug hunting scenarios. Current evaluation methods suffer from three primary gaps:

1. Fuzz-Harness Dependence: Many benchmarks (e.g., ARVO, CyberGym) rely on fuzzing harnesses, which narrow the executable entry point to specific internal interfaces. This measures harness-guided input mutation rather than the ability to audit code through public interfaces.
2. Target-Specific or Pattern-Based Grading: Existing metrics often compare generated evidence against known patches, exit codes, or crash signatures. This conflates the reproduction of the intended bug with unrelated crashes in the same historical revision or rejects valid PoCs that produce different traces.
3. Excessive Information Exposure: Task inputs in current benchmarks often include sanitizer reports, function names, line numbers, or triggering paths. This removes the uncertainty security engineers face when auditing code, providing agents with information they would not possess in a real-world scenario.

Consequently, there is a need for a benchmark that measures agent bug hunting on critical, high-complexity software systems under realistic conditions, requiring reasoning across execution tiers without reliance on fuzzing infrastructure or over-exposed metadata.

Methodology

The authors present SEC-bench Pro, a benchmark and construction pipeline designed to measure agent bug hunting on JavaScript engines (specifically V8 and SpiderMonkey). The system employs a three-phase, self-evolving pipeline to convert disclosed security reports into reproducible, validated tasks.

1. Report Collection

The pipeline ingests structured security reports from project-specific trackers (Chromium Issue Tracker, Mozilla Bugzilla) and curated advisory feeds (MFSA, Pwn2Own, CISA KEV). It filters for reports labeled sec-high or sec-critical (or bounty-qualified) that include both a concrete PoC artifact and a linked fix commit.

2. Agent-Driven Environment Reconstruction

To address the labor intensity of reconstructing historical vulnerable environments, the pipeline delegates this task to autonomous coding agents.

• Input: Agents receive the raw bug report, the PoC artifact, and a structured prompt specifying the target binary, allowed flags, and expected error types.
• Process: Agents drive the project's build system to produce instrumented binaries at the specific reported revision, configured with appropriate sanitizers (e.g., ASan) or debug checks.
• Output: The agent generates a Docker artifact bundle containing the vulnerable and fixed Dockerfiles, structured metadata, the canonical PoC, and captured crash signatures. The prompt explicitly forbids fabricated outputs.

3. Automated Validation Oracles

Before an instance enters the dataset, it must pass two automated oracles:

• Vulnerable-Image Oracle: Runs the PoC multiple times in the reconstructed container. It classifies crashes against a project-specific taxonomy (e.g., ASAN_CRASH, SANDBOX_VIOLATION, DCHECK). An instance passes only if the execution produces a crash matching the expected error type.
• Fixed-Image Oracle: Applies the linked patch, rebuilds the binary, and runs the same PoC. It verifies that the patch suppresses the collected PoC (classifying outcomes as BLOCKED_DEFENSIVE, BLOCKED_HARMLESS, or BLOCKED_NO_REPRO). An instance passes only if the PoC no longer reproduces after the fix.

Evaluation and Grading

The evaluation uses a three-image grading pipeline (vulnerable, fixed, and latest upstream images) and an LLM-as-a-Judge to determine success.

• Execution: Candidate PoCs are replayed on all three images.
• Judgment: The LLM judge analyzes exit codes, stdout, and stderr to classify the outcome into three categories:
  • Verified: The vulnerable image shows a crash matching the target; fixed/latest evidence does not contradict this attribution.
  • Unsure: Evidence is incomplete (e.g., infrastructure failure in fixed/latest runs), requiring manual adjudication.
  • Illegal: The crash does not match the target, or the fixed/latest images show the crash persists due to a different root cause.
• Success Metric: An instance is successful if at least one candidate PoC receives a verified outcome. This approach prevents "crash-only" inflation where agents trigger unrelated crashes.

Key Contributions

• SEC-bench Pro Dataset: A collection of 183 validated vulnerabilities across V8 (103 instances) and SpiderMonkey (80 instances). The V8 subset includes reports with cumulative Google Vulnerability Reward Program awards exceeding $1.5 million.
• Self-Evolving Pipeline: A parameterized construction pipeline that can ingest new PoC-backed and patch-backed reports without redesigning the benchmark, applicable to any project with source-level build recipes and public reports.
• Three-Image Grading Oracle: A robust evaluation method that distinguishes between intended bug reproduction and unrelated crashes or infrastructure failures, addressing the limitations of crash-only or patch-matching graders.
• Comprehensive Agent Evaluation: An evaluation of three state-of-the-art coding agent scaffolds (CODEX, CLAUDECODE, OPENCODE) on long-horizon security tasks.

Results

The evaluation of frontier models and open-weight baselines reveals significant limitations in current capabilities for long-horizon bug hunting:

• Low Success Rates: Even the strongest configurations remain below 40% success on both engines.
  • V8: The strongest single configuration (CODEX with GPT-5.4) achieves 32.0% (33/103). The open-weight Kimi-K2.6 baseline achieves 11.7%.
  • SpiderMonkey: The strongest configuration (CLAUDECODE with Opus 4.6) achieves 38.8% (31/80).
• Complementary Coverage: The agents solve largely disjoint sets of instances. The union of CLAUDECODE and CODEX reaches 37.9% on V8 and 48.8% on SpiderMonkey, suggesting that no single agent strategy dominates all vulnerability classes.
• Strategy Differences:
  • CLAUDECODE employs a high-breadth strategy, generating many speculative PoCs with low per-PoC yield. It often fails because the PoCs never trigger a crash (clean exit codes) or trigger unrelated crashes.
  • CODEX employs a high-selectivity strategy, generating fewer but higher-confidence PoCs. It often fails because it abstains from submitting when it cannot locally verify reachability, or its PoCs trigger crashes that are not fully mitigated by the patch (partial-fix collisions).
• Cost of Failure: Failed runs consistently consume significantly more tokens and runtime than successful runs (e.g., a 3.0x token gap for CLAUDECODE on V8), indicating that prolonged exploration often leads to diminishing returns rather than deeper analysis.
• Importance of Attribution: A "crash-only" grader would inflate success rates by approximately 43.6% across configurations, highlighting the necessity of the three-image LLM judge to filter out mis-attributed crashes.

Significance

The paper establishes SEC-bench Pro as a concrete, reproducible target for measuring progress in automated software security. It demonstrates that while LLMs can assist in vulnerability discovery, current frontier models struggle with the complexity of long-horizon bug hunting on production-grade engines. The benchmark exposes specific limitations, including the difficulty of synthesizing complex trigger conditions (e.g., specific JIT states, sandbox bypasses) and the high cost of failed exploration. The authors conclude that future systems require stronger trigger synthesis, more robust attribution mechanisms, and ensemble search strategies to close the gap between current capabilities and the requirements of real-world security auditing.