LLM-based Vulnerability Detection at Project Scale: An Empirical Study

Technical Summary: LLM-based Vulnerability Detection at Project Scale: An Empirical Study

Problem Statement
As software complexity and deployment scales increase, security vulnerabilities have become more frequent and harmful, necessitating robust automated detection. While traditional static analysis tools are widely adopted, they rely on manually engineered rules and predefined patterns, limiting their ability to generalize to unseen or evolving vulnerability types. Conversely, recent Large Language Model (LLM)-based detectors promise enhanced semantic reasoning by interpreting code semantics and complex logic without explicit rule engineering. However, most existing LLM-based approaches are evaluated on function-level or hunk-level snippets, leaving their effectiveness, reliability, and scalability at the project scale unclear. There is a lack of systematic understanding regarding the root causes of failures (specifically false positives) and the computational overhead when these specialized detectors are applied to large, real-world codebases.

Methodology
The authors present the first comprehensive empirical study comparing five specialized, project-scale LLM-based vulnerability detectors against two traditional static analyzers (CodeQL and Semgrep). The study employs a dual-setting evaluation strategy:

1. In-House Benchmark: A curated dataset of 222 known real-world vulnerabilities (covering 8 CWE types in C/C++ and Java) derived from ReposVul, CWE-Bench-Java, and JLeaks. This setting assesses detection capability (Recall) against ground truth.
2. Real-World Projects: A set of 24 actively maintained open-source projects (including Linux kernel components, ImageMagick, Jenkins, and Keycloak). This setting evaluates practical usability by analyzing warning volumes and False Discovery Rates (FDR).
3. Manual Analysis: Two security experts manually inspected 385 sampled warnings (requiring over 150 human hours) to construct a taxonomy of false positive root causes.
4. Cost Measurement: The study quantified computational overhead, measuring input/output token consumption and end-to-end runtime for each tool.

The evaluated LLM-based methods include RepoAudit, KNighter, IRIS, LLMDFA, and INFERROI, representing diverse workflows such as multi-agent frameworks, LLM-assisted checker generation, and control-flow guided inference.

Key Contributions

• First Project-Scale Comparison: The study provides the first systematic comparison of five representative LLM-based detectors against traditional tools at the project scale.
• Comprehensive Evaluation: It evaluates detection effectiveness on 222 known vulnerabilities and practical usability across 24 real-world projects.
• Failure Taxonomy: The authors manually label 385 reports to construct a detailed taxonomy of false positive causes, quantifying how design choices (e.g., dataflow construction, source/sink inference) contribute to failures.
• Cost Analysis: The paper measures and reports the substantial token usage and time costs associated with project-scale LLM-based detection.
• Artifact Release: All experimental artifacts, including scripts, prompts, and taxonomy labels, are released to foster reproducibility.

Key Results

1. Low Recall but Unique Detections: On the in-house benchmark, LLM-based methods exhibited low average recall (21.09% for C/C++ and 33.82% for Java), often missing vulnerabilities due to misaligned source/sink definitions. However, they still uncovered more unique vulnerabilities than traditional tools, suggesting complementary value.
2. High False Discovery Rates (FDR): In real-world projects, both LLM-based and traditional tools generated substantial warnings with very high FDRs. Even the best-performing tool had an average FDR of 85.3%. For instance, RepoAudit on C/C++ projects had an average SFDR of 97.0%.
3. Root Causes of Failure: The manual analysis identified two dominant causes shared by both LLM and traditional tools:
   • Shallow Interprocedural Reasoning (37.47% of FPs): Tools fail to trace dataflows through deep call chains or complex control structures.
   • Imprecise Source/Sink Identification (19.00% of FPs): Tools misidentify benign program points as security-relevant or fail to recognize project-specific APIs.
   • LLM-Specific Failures: LLM-based tools additionally suffered from hallucinated code understanding, incorrect control flow reconstruction (e.g., merging mutually exclusive paths), and failure to adhere strictly to prompt definitions or CWE scopes.
4. Computational Overhead: Project-scale LLM detection is computationally expensive. Tools like RepoAudit and LLMDFA required hundreds of thousands to hundreds of millions of tokens per project, with runtimes ranging from multi-hour to multi-day (e.g., up to 4,638 minutes). Traditional tools (CodeQL, Semgrep) completed analyses in seconds to minutes with negligible token costs.

Significance and Implications
The paper concludes that while LLM-based detectors offer promising semantic reasoning capabilities that can uncover vulnerabilities missed by traditional pattern-matching tools, they are not yet mature enough for reliable, standalone deployment in real-world scenarios. The study highlights critical limitations in robustness (high FDR), reliability (low recall on ground truth), and scalability (prohibitive computational costs).

The authors derive five implications for future research:

1. Deepen Dataflow Reasoning: Moving beyond shallow call-graph exploration to holistic "input → propagation → output" analysis.
2. Improve Source/Sink Inference: Developing context-aware mechanisms to accurately identify project-specific APIs rather than relying on generic or hallucinated definitions.
3. Enhance Semantic Understanding: Integrating LLMs with intermediate program representations and static analysis signals to better handle complex language features (e.g., destructors, try-with-resources).
4. Enforce Prompt Alignment: Implementing structured multi-agent collaboration or self-verification to ensure models strictly adhere to vulnerability definitions.
5. Address Scalability: Adopting hierarchical analysis, caching intermediate reasoning, and optimizing workflows to reduce token consumption and runtime to levels suitable for continuous integration.

The authors emphasize that current findings underscore the need for hybrid approaches that leverage LLM semantic strengths while mitigating their cost and reliability issues through rigorous static analysis integration.