Technical Summary: Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents

Problem Statement

The detection of Vulnerability-Inducing Commits (VICs) at submission time, known as Just-In-Time Vulnerability Detection (JIT-VD), is critical for securing software supply chains. However, this task is highly challenging because it requires reasoning about the semantic impact of code changes across heterogeneous information sources, including code diffs, commit messages, and surrounding contextual code.

Existing approaches, ranging from feature-based methods to deep learning models (e.g., neural networks, graph-based models, and transformers), often struggle to fully capture complex commit semantics and developer intent. While recent Large Language Model (LLM) based approaches and agent frameworks (e.g., CodeAgent, ReAct) have shown promise in code understanding, they are primarily designed for general vulnerability detection or code review. They often lack specialized mechanisms for reasoning about VICs in evolving codebases. Furthermore, iterative reasoning loops (like ReAct) can lead to error accumulation and lack explicit mechanisms for result verification, limiting transparency and reliability.

Methodology: VIC-RAGENT

The authors propose VIC-RAGENT, an LLM-based multi-agent framework designed for effective and explainable vulnerability detection. The system employs a structured multi-stage reasoning pipeline that decomposes the detection task into role-specialized components and progressive refinement stages.

Framework Architecture

The framework operates through four sequential stages, utilizing specific agents to process inputs (commit message, code diff, file-level context):

1. Stage 0: Preparation

   • Code Analyst (CA): Extracts structural information (e.g., simplified call graphs, data flow patterns) from the code diff and context to understand how modified code interacts with the system.
   • Target Analyst (TA): Interprets the developer's intent by analyzing the commit message and aligning it with the modified files. It categorizes changes (e.g., bug fixes, refactoring) and identifies inconsistencies between goals and implementation that may indicate risks.

2. Stage 1: Preliminary Inspection (Candidate Generation)

   • Vulnerability Inspector (VI): Acts as a high-recall filter. Guided by the structural and intent representations from Stage 0, it scans modified code fragments against six vulnerability categories (I/O Validation, Memory Safety, Web Security, Authentication/Authorization, Resource Management, File/Path Handling).
   • This stage intentionally relaxes verification constraints to generate a "superset" of plausible vulnerability hypotheses, prioritizing recall to ensure subtle risks are not prematurely discarded.

3. Stage 2: Reanalysis (Type-Guided Verification)

   • Vulnerability Inspector (VI): Refines the candidates from Stage 1 through constrained reasoning. It re-evaluates each fragment based on its predicted vulnerability type.
   • Knowledge-Augmented Reasoning: The system utilizes a Vulnerability Knowledge Base containing historical security reports. Using Retrieval-Augmented Generation (RAG), it retrieves similar historical cases to provide evidence for case-based reasoning. If no similar cases exist, it relies on contextual information.

4. Stage 3: Final Decision (Verification + Aggregation)

   • Vulnerability Inspector (VI): Performs a conservative, low-temperature verification of the refined hypotheses.
   • Aggregation: A commit is classified as vulnerable if at least one fragment is confirmed.
   • Document Specialist (DS): If a commit is confirmed as vulnerable, this agent generates a structured security report stored in the knowledge base for future use.
   • Audit Supervisor (AS): Monitors agent outputs to ensure consistency with predefined requirements throughout the process.

Implementation Details

• LLMs: Evaluated using DeepSeek-V3.2, Qwen-Plus, and GPT-4o-mini.
• Temperature Settings: Varies by stage to balance creativity and determinism (e.g., 0.4 for preliminary inspection to encourage candidate generation, 0.1 for final decision to ensure conservative judgment).
• Knowledge Base: Organized by vulnerability types with vector indices (using CodeBERT embeddings) for similarity-based retrieval. It includes a maintenance mechanism to periodically remove false positive (FP) reports to prevent error propagation.

Key Contributions

1. Multi-Agent Framework for JIT-VD: Introduction of VIC-RAGENT, which analyzes potential vulnerabilities by integrating structural analysis, intent understanding, and vulnerability inspection through specialized agents.
2. Multi-Stage Reasoning Process: A structured workflow (Preliminary Inspection $\rightarrow$ Reanalysis $\rightarrow$ Final Decision) that enables coarse-to-fine reasoning, improving detection reliability.
3. Knowledge-Augmented Reasoning: Design of a vulnerability knowledge base that allows the system to leverage historical vulnerability cases via retrieval-augmented reasoning during the reanalysis stage.
4. Empirical Evaluation: Comprehensive experiments on a real-world dataset (V-SZZ) demonstrating the framework's performance across multiple LLMs.

Experimental Results

The framework was evaluated on the V-SZZ dataset (241 commits: 106 VICs, 135 VFCs) and compared against three baselines: Direct (vanilla prompting), Chain-of-Thought (CoT), and CodeAgent.

• Performance: VIC-RAGENT consistently outperformed all baselines across three different LLMs.
  • Compared to the strongest baseline (CodeAgent), VIC-RAGENT achieved 1.2–1.7× higher F1-scores.
  • It demonstrated significant improvements in Recall, achieving up to 2× higher recall than the strongest baseline. For instance, with GPT-4o-mini, VIC-RAGENT achieved 75% recall compared to 60% for CodeAgent.
• Ablation Study:
  • Removing Stage 2 (Reanalysis) reduced recall and F1-score, indicating its importance in coverage and disambiguating suspicious fragments.
  • Removing Stage 3 (Final Decision) increased recall but significantly increased False Positives (FP), demonstrating that Stage 3 acts as a necessary conservative filter to improve prediction trustworthiness.
• Generalization: When tested on recent CVEs (unseen during training), VIC-RAGENT maintained superior performance, achieving the highest recall (55%) and F1-score (60%) among all methods, suggesting strong generalization capabilities.
• Cost: VIC-RAGENT incurs higher computational costs (approx. 0.0487 USD per commit vs. 0.0079 for Direct) due to multi-stage reasoning, but the authors argue this is justified for offline or CI-integrated security auditing where accuracy is prioritized over latency.

Significance and Claims

The paper claims that VIC-RAGENT offers a robust, explainable, and practical solution for detecting VICs in modern software development workflows. Its primary significance lies in addressing the limitations of existing single-pass or general-purpose agent approaches by introducing a structured, multi-stage reasoning process that explicitly separates reasoning objectives.

The authors emphasize that the framework's ability to balance exploration (high recall in Stage 1) with verification (conservative filtering in Stage 3), combined with knowledge-augmented reasoning, allows it to detect subtle vulnerabilities that other methods miss. While acknowledging limitations such as reliance on LLM consistency and the need for manual FP cleanup in the knowledge base, the work demonstrates that decomposing the vulnerability detection task into specialized roles and stages significantly improves detection performance and reliability.