MAS-SZZ: Multi-Agentic SZZ Algorithm for Vulnerability-Inducing Commit Identification

1. Problem Statement

Accurate identification of Vulnerability-Inducing Commits (VICs) is critical for software security tasks such as vulnerability detection, severity assessment, and affected version analysis. The standard approach, the SZZ algorithm, traces code changes backward from a Vulnerability-Fixing Commit (VFC) to find the original commit that introduced the bug.

However, existing SZZ algorithms (including foundational variants like B-SZZ, AG-SZZ, and state-of-the-art LLM-based approaches like LLM4SZZ) suffer from two primary limitations:

1. Incorrect Anchor Selection: Traditional methods rely on simple diff-based heuristics (e.g., deleted/modified lines) to select "anchor statements" for backtracking. This fails when vulnerabilities are hidden in unmodified context or when "composite" and "ghost" commits introduce noise, leading to high false-positive rates. Even LLM4SZZ struggles because it cannot always distinguish between security fixes and non-security changes (e.g., refactoring).
2. Inadequate Backtracking Capability: Standard algorithms often perform single-step backtracking or use similarity-based heuristics that stop prematurely (over-tracing) or miss the true origin (under-tracing). They lack the dynamic reasoning required to determine the exact moment a vulnerability was introduced.

2. Methodology: MAS-SZZ

The authors propose MAS-SZZ, a Multi-Agentic SZZ system that leverages Large Language Models (LLMs) and autonomous agents to overcome these challenges. The system operates in three distinct stages, orchestrated by specialized agents (Auditor, Judge, Reviewer, Evaluator, Locator, and Tracer).

Stage 1: Evidence-Grounded Root Cause Analysis

• Input: CVE description and the corresponding VFC (commit message and code diff).
• Process:
  • The Auditor Agent analyzes the inputs to summarize the vulnerability's root cause. Crucially, it generates evidence points linking every claim back to specific parts of the CVE or the code diff to prevent hallucinations.
  • The Judge Agent performs a reliability check on two dimensions:
    1. Evidence Traceability: Can the evidence be mapped to the provided inputs?
    2. Logical Consistency: Is the root cause consistent with the commit message?
  • If the check fails, the Judge provides corrective feedback to the Auditor for a targeted re-generation (up to 3 rounds).

Stage 2: Intent-Driven Anchor Statement Selection

• Goal: Filter out non-security changes (e.g., refactoring, chores) to identify the specific lines responsible for the vulnerability.
• Process:
  • The Reviewer Agent uses a structured step-forward prompting strategy (Chain-of-Thought) to analyze each patch hunk. It infers the change intent and classifies it (e.g., "fix," "refactor," "chore") based on the Conventional Commits Specification.
  • The Evaluator Agent checks if the intent of a "fix" hunk aligns with the previously established root cause.
  • The Locator Agent selects the specific anchor statement(s) from the relevant hunks, considering the code change, surrounding context, and root cause.

Stage 3: Autonomous Repository Exploration

• Goal: Trace backward through the repository history to find the first commit containing the vulnerability.
• Process:
  • The Tracer Agent starts from the anchor statement and iteratively executes git blame to move backward through the commit history.
  • At each step, the agent analyzes the commit message and the code state (using tools like ExpandContext and LocateSymbol) to determine if the vulnerability exists in that revision.
  • Dynamic Stopping Criteria: Unlike fixed-step algorithms, the Tracer stops when the vulnerability is no longer found in the current commit but was present in the previous one. The last commit where the vulnerability existed is identified as the VIC.

3. Key Contributions

1. Multi-Agent Architecture: Designed MAS-SZZ, a novel system where specialized agents collaborate to handle root cause analysis, intent identification, and autonomous history exploration.
2. Optimization Strategies:
   • Evidence-Grounded Analysis: Ensures root cause summaries are verifiable against inputs, reducing hallucinations.
   • Intent-Driven Selection: Uses LLMs to distinguish security fixes from noise, solving the anchor selection challenge.
   • Autonomous Exploration: Replaces static heuristics with dynamic, iterative backtracking guided by the root cause.
3. Comprehensive Evaluation: Validated the approach against seven existing SZZ algorithms on two human-annotated benchmark datasets (V-SZZ and Java-SZZ) covering C/C++ and Java.

4. Experimental Results

The authors evaluated MAS-SZZ using Gemini-3.1-pro as the backbone LLM against baselines including B-SZZ, AG-SZZ, L-SZZ, R-SZZ, MA-SZZ, V-SZZ, and LLM4SZZ.

• Performance: MAS-SZZ outperformed all baselines across all datasets in terms of F1-score.
  • It achieved a 65.22% relative improvement in F1-score over the best-performing baseline on the Java-SZZ dataset.
  • It achieved top-2 performance in 8 out of 9 evaluation metric combinations (Precision, Recall, F1 across three datasets).
• Ablation Study:
  • Removing Anchor Selection (w/o AS) caused significant drops in Precision (up to 59.52%), proving the necessity of filtering non-relevant code hunks.
  • Removing Agentic Backtracking (w/o AB) caused a massive drop in F1-score (up to 54.17%), demonstrating that autonomous, iterative exploration is superior to static, single-step tracing.
• LLM Selection: Heavyweight models (e.g., Gemini-3.1-pro) generally outperformed lightweight ones, with Gemini-3.1-pro achieving the best overall results.

5. Significance

MAS-SZZ represents a paradigm shift in vulnerability analysis by moving from heuristic-based line mapping to semantic-based, multi-agent reasoning.

• Reliability: By grounding root cause analysis in evidence and using intent-driven filtering, it significantly reduces false positives caused by composite or ghost commits.
• Adaptability: The autonomous backtracking mechanism allows the system to handle complex version histories where traditional similarity metrics fail.
• Practical Impact: The substantial gains in accuracy (up to 65% improvement) suggest that MAS-SZZ can be reliably deployed in real-world software supply chain security tools to accurately identify the origin of vulnerabilities, enabling faster remediation and better risk assessment.