Technical Summary: ALIBI - Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

Problem Statement

As Large Language Models (LLMs) are increasingly deployed for security-sensitive tasks like vulnerability detection and code review, they rely heavily on natural-language context embedded within source code (e.g., comments, documentation, API contracts) to reason about program intent and verify safety claims. This reliance creates a previously underexplored attack surface: adversarial source-code comments.

Unlike traditional attacks that modify executable code to hide vulnerabilities, this paper investigates a "reasoning failure" mode. The core problem is whether an adversary can synthesize a new, genuinely vulnerable implementation of a coding task and strategically insert semantically coherent but false safety claims into source-code comments. These comments aim to convince the LLM-based detector to override its correct analysis of the executable code and accept the false claim, thereby classifying a vulnerable program as benign without altering the program's behavior or the vulnerability itself.

Methodology: The ALIBI Framework

The authors present ALIBI (Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments), an automated, adaptive black-box attack framework. The framework operates through a three-phase pipeline:

1. Code Synthesis:

   • The attacker receives a natural-language coding task derived from real-world vulnerability-fixing commits (specifically C/C++ null-pointer dereferences, CWE-476).
   • An attacker model (QWEN3.6-27B-FP8) synthesizes a functional implementation that intentionally contains a reachable null-pointer dereference.
   • The implementation is validated to ensure it compiles and passes the repository's existing test suite, ensuring the attack targets code that would survive standard development checks.

2. Reasoning-Guided Initialization:

   • The vulnerable code is submitted to the target detector.
   • If the detector correctly identifies the vulnerability, the attacker analyzes the detector's reasoning trace to identify the specific evidence supporting the "vulnerable" verdict.
   • The attacker then fabricates an unverifiable safety claim targeting the gap in the detector's reasoning (e.g., claiming an external invariant guarantees pointer safety).
   • This initial comment is generated using one of 10 presentation strategies, ranging from simple notes (NOTE, USR) to authoritative claims citing external tools (TOOL_COVERITY, TOOL_CLANGSA, TOOL_FRAMA), call graphs (CALL_GRAPH), or step-by-step security analyses (COT).

3. Adaptive Refinement:

   • If the initial comment fails to flip the verdict, ALIBI enters an iterative refinement loop (up to 5 rounds).
   • The attacker uses the detector's feedback (verdict and reasoning) to refine the comment's content and placement.
   • This process continues until the detector's verdict flips to "Benign" or the query budget is exhausted.

Benchmark Construction:
The authors constructed a benchmark by transforming 184 real-world vulnerability-fixing commits (from the MegaVul dataset) into coding tasks. After filtering for build success, test suite passage, and independent verification of the injected vulnerability, the final benchmark consists of 125 validated null-pointer dereference instances.

Key Contributions

1. Identification of a New Attack Surface: The paper identifies that non-executable source-code comments can be used to manipulate LLM-based vulnerability detectors, exploiting a "comment trust prior" where detectors assign undue evidential weight to plausible but unverified safety claims.
2. ALIBI Framework: The design of an adaptive black-box framework that generates semantically tailored adversarial comments and iteratively refines them using detector feedback.
3. Novel Benchmark: A benchmark that transforms vulnerability-fixing commits into coding tasks to evaluate detectors on newly synthesized vulnerable implementations, rather than historical vulnerable code.
4. Comprehensive Evaluation: Evaluation of ALIBI against four representative LLM-based detectors (OPENVUL, VULNLLM-R, VUL-RAG, VULTRIAL) and various defense strategies.
5. Defense Analysis: An evaluation showing that prompt-level defenses are limited, while architectural isolation and pre-detector sanitization offer substantial resilience.

Experimental Results

Attack Effectiveness (RQ1)

ALIBI achieved extremely high success rates across all four evaluated detectors:

• OPENVUL: 97.4% Attack Success Rate (ASR) (112/115 cases).
• VULNLLM-R: 91.2% ASR (103/113 cases).
• VUL-RAG: 95.5% ASR (106/111 cases).
• VULTRIAL: 100.0% ASR (105/105 cases).

The attacks succeeded by replacing the detector's correct reasoning with fabricated external evidence (e.g., "Frama-C verified," "AST builder guarantees") that the detector could not verify from the visible code.

Presentation Strategy (RQ2)

The effectiveness of the attack heavily depended on the presentation style:

• Most Effective: COT (Chain-of-Thought reasoning) achieved the highest ASR (81.1%). Strategies invoking authoritative external evidence (e.g., TOOL_COVERITY, CALL_GRAPH, TOOL_FRAMA) were also highly effective (>70% ASR).
• Least Effective: Strategies resembling informal developer notes (USR, ASSERT, FAKE_TEST) performed worse than the baseline NOTE strategy, as detectors explicitly treated them as less trustworthy.

Attack Efficiency (RQ3)

• Diversity: No single presentation style covered all vulnerabilities. A portfolio of diverse strategies (e.g., COT + TOOL_COVERITY + TOOL_FUZZER) was required to maximize coverage.
• Refinement: Adaptive refinement significantly improved success rates. Approximately 30 detector queries were sufficient to achieve >90% coverage across all systems.

Defense Evaluation (RQ4)

• Prompt-Level Mitigations (D1, D2): Instructing the model to treat comments as untrusted provided limited protection. Attackers successfully adapted by steering the model's reasoning even while the model acknowledged the comment was untrusted.
• Architectural Isolation (D3): Running a code-only baseline analysis in a separate call before introducing comments substantially improved robustness (e.g., reducing ASR to 0% on VULNLLM-R), though it was less effective on multi-stage debate systems like VULTRIAL.
• Pre-Detector Sanitization (D4): Filtering comments before inference (removing unverifiable claims) provided the strongest protection, reducing ASR to below 3% across all systems.

Generalization (RQ6)

The framework was tested on Use-After-Free (UAF) vulnerabilities (CWE-416). The attack generalized successfully, achieving >95% ASR on three of the four detectors, confirming that the vulnerability lies in the reasoning mechanism regarding external claims, not just the specific bug type.

Significance and Claims

The paper claims that current LLM-based vulnerability detectors possess a fundamental weakness: they lack the ability to rigorously distinguish between evidence in the code and plausible but unverifiable claims in natural language.

• Trust Prior: Detectors exhibit a "comment trust prior," where they defer to safety claims attributed to external authorities (static analyzers, framework invariants) even when those claims cannot be verified from the submitted source code.
• Design Implication: The authors argue that robustness to adversarial software documentation must be a primary design objective. They conclude that architectural isolation and pre-detector sanitization are necessary to decouple natural-language context from program evidence, rather than relying on prompt engineering alone.
• Scope: The work is framed as an evaluation of robustness using a reproducible benchmark, highlighting a specific failure mode in the integration of LLMs into security workflows. It does not claim to solve the problem but rather motivates security-aware designs that carefully calibrate trust between natural-language context and program evidence.