Technical Summary: Snyk VulnBench JS 1.0

Problem Statement

As coding agents and Large Language Models (LLMs) integrate into the software development lifecycle to perform security reviews, a critical reliability question arises: Is agentic LLM security review repeatable? Unlike traditional Static Application Security Testing (SAST) tools, which are deterministic and produce identical outputs for identical inputs, LLMs are probabilistic. They may vary in their findings across runs, over-report adjacent concerns, or stop after identifying a single instance of a repeated pattern.

The paper addresses the lack of empirical data quantifying this variability. It seeks to determine if an LLM agent, when presented with the same vulnerable JavaScript code and prompt multiple times, will report the same security issues consistently.

Methodology

The authors introduced Snyk VulnBench JS 1.0, a benchmark designed to measure repeatability and agreement under controlled conditions.

• Dataset: The benchmark consists of 10 small JavaScript and Express-based application fixtures. These range from single-file snippets to a larger "todo" application (approx. 380 lines) featuring server routes, database interactions, file uploads, and frontend logic.
• Reference Standard: Snyk Code (a deterministic SAST tool) serves as the reference set. The benchmark does not claim Snyk Code identifies all possible vulnerabilities; rather, it uses Snyk Code's ability to reproducibly identify its own reference findings as the baseline for measurement.
• Experimental Design:
  • Models: Six configurations were tested using the Claude Code harness and TypeScript Agent SDK:
    • Claude Opus 4.6 (Medium, High)
    • Claude Opus 4.7 Max
    • Claude Sonnet 4.6 (Medium, High)
  • Repetitions: Each of the 10 tasks was run 5 times per configuration, totaling 300 runs.
  • Constraints: Models received the same audit prompt and project files but were explicitly prevented from reading the findings.json reference file.
• Scoring Mechanism:
  • Snyk-reference F1: The primary metric is the harmonic mean of precision and recall, treating Snyk Code findings as the ground truth.
  • Leniency: A model finding is credited if it reports the same vulnerability type as a reference finding, regardless of exact file, line, severity, or source-to-sink path.
  • Stability Metrics: The study tracks how often specific findings (both matched and unmatched) appear across the five repetitions.

Key Contributions

1. Quantification of LLM Variance: The paper provides the first systematic measurement of how LLM security findings vary across identical runs, distinguishing between "reference-matched" stability and "extra" report noise.
2. Demonstration of Complementarity: It empirically shows that LLMs and deterministic SAST tools fail in different ways, suggesting they are complementary rather than interchangeable.
3. Cost vs. Performance Analysis: The study evaluates whether more expensive, larger models yield better security coverage or stability.
4. Identification of Product Gaps: The benchmark methodology successfully surfaced potential gaps in the SAST reference tool (Snyk Code) where LLMs identified valid vulnerabilities that the SAST tool missed.

Results

1. Repeatability and Stability

• Reference-Matched Findings: When LLMs identified vulnerabilities already found by Snyk Code, they were generally stable. 84.8% of unique reference-matched findings appeared in all five repetitions.
• Unmatched Findings (The "Noise"): Findings not present in the Snyk Code reference set were highly unstable.
  • 49.7% of unique unmatched findings appeared in only one of the five runs.
  • Only 13.7% of unmatched findings appeared in all five runs.
  • Model Variance: Claude Sonnet 4.6 Medium produced the most unstable "extra" reports, with 61.7% of its unmatched findings appearing in only one run. In contrast, Claude Opus 4.6 Medium was the most stable, with 0% of its unmatched findings appearing in just one run.

2. Coverage and Performance

• F1 Scores: The best-performing LLM configuration (Claude Opus 4.6 Medium) achieved a 75.4% Snyk-reference F1 score, leaving a 24.6-point gap compared to the deterministic SAST baseline (100.0%).
• Vulnerability Class Performance:
  • Strengths: LLMs performed well on familiar, high-signal exploit shapes such as command injection, code injection, hardcoded credentials, SQL injection, SSRF, and prototype pollution.
  • Weaknesses: LLMs struggled with systematic SAST classes, including resource-limit findings, improper sanitization, type validation, insecure transport, and repeated path traversal flows.
• Case Study (js-project-nightowl): In the largest fixture, the best model (Claude Opus 4.6 High) achieved only 40.0% Snyk-reference F1. It consistently missed path traversal and resource-limit vulnerabilities, failing to enumerate repeated data-flow sinks where deterministic analysis excelled.

3. Cost vs. Quality

• Diminishing Returns: The most expensive model, Claude Opus 4.7 Max, cost 5.7x more and used 1.9x more tokens than the best-performing model (Claude Opus 4.6 Medium).
• Performance: Despite the higher cost, Opus 4.7 Max scored lower (68.8% F1) than Opus 4.6 Medium (75.4% F1).
• Conclusion: Higher inference cost does not automatically translate to better security coverage or stability.

4. Complementarity and Product Gaps

The study identified specific instances where LLMs found valid issues that SAST missed:

• False Positives: Some LLM reports were "vulnerability-shaped" but not exploitable (e.g., a SQL injection report on a mock helper function that returns an empty array).
• True Positives (Product Gaps): In the js-project-nightowl fixture, LLMs reported SQL injection vulnerabilities that were not in the Snyk Code reference set. The authors argue these are likely valid findings representing a gap in the SAST tool's coverage, rather than hallucinations.

Significance and Claims

The paper concludes that agentic LLM security review is not uniformly repeatable. While LLMs can effectively identify familiar, high-signal vulnerabilities, their "extra" reports are often noisy and inconsistent.

• Not a Replacement: The results argue against treating LLMs as a replacement for deterministic SAST. SAST is superior for systematically enumerating repeated data-flow sinks and is deterministic.
• Complementary Approach: The optimal strategy is a combined workflow. Deterministic SAST provides a stable baseline for known patterns, while agentic LLMs can surface adjacent concerns and potential product gaps (like the SQL injection in js-project-nightowl) that static analyzers miss.
• Future Work: The authors acknowledge the limitations of the current benchmark (small snippets, lack of independent ground truth) and propose future releases to include full-fledged applications, business-logic vulnerabilities, and independent adjudication sources (e.g., BaxBench-style data).

The paper emphasizes that the goal is not to declare a "winner" in a leaderboard sense, but to measure the repeatability, variance, and complementarity of these technologies to inform how they should be integrated into secure development loops.