Security Is Relative: Training-Free Vulnerability Detection via Multi-Agent Behavioral Contract Synthesis

1. Problem Statement: The Semantic Ambiguity Crisis

The paper identifies a critical failure in current automated vulnerability detection systems, particularly when evaluated on rigorous benchmarks like PrimeVul.

• Performance Collapse: Models achieving high F1 scores (>0.68) on legacy datasets (e.g., Big-Vul) degrade to near-random guessing (F1 < 0.031) under strict deduplication and temporal splitting.
• Root Cause (Semantic Ambiguity): The authors argue that vulnerability is not an absolute property of code syntax. Identical code patterns can be secure in one project context and vulnerable in another, depending on specific behavioral contracts (e.g., middleware sanitization, architectural conventions).
• The "Double Standard" Problem: Analysis of the PrimeVul dataset reveals that code with >99% similarity often receives opposite vulnerability labels depending on the specific CVE context. Existing models fail because they rely on global pattern matching rather than understanding project-specific security intents.
• Limitations of Current Approaches:
  • Retrieval-Augmented Generation (RAG): Relies on historical similarity, failing on zero-day logic flaws.
  • Agentic Static Analysis: Limited by fragile domain-specific languages (e.g., CodeQL) and compilation dependencies.
  • Context-Enriched Reasoning: Often degrades performance due to attention dilution when models lack explicit guidance on what to verify.

2. Methodology: The Phoenix Framework

The authors propose Phoenix, a training-free, zero-shot multi-agent framework that reframes vulnerability detection from an open-ended classification task to a closed-form contract verification task. The system utilizes three specialized agents operating sequentially:

A. Agent 1: Semantic Slicer

• Function: Extracts the minimal, vulnerability-relevant code context from a paired sample (vulnerable code + patched code).
• Mechanism: It identifies the specific "diff" (patch location), consults the CVE description/commit message for root causes, and ruthlessly prunes boilerplate.
• Impact: Reduces average function length by 82.8% (from ~5,300 to ~900 characters), minimizing cognitive load for downstream agents.

B. Agent 2: Requirement Reverse Engineer

• Function: Synthesizes a Gherkin behavioral specification (a structured Given-When-Then format) that encodes the precise security contract distinguishing the vulnerable code from the patched code.
• Mechanism: It reverse-engineers the implicit security rule (e.g., "If input is null, return error") and formalizes it into a definitive test scenario.
• Innovation: This acts as an explicit intermediate representation, externalizing the "security contract" so the judge does not need to infer it from raw code.

C. Agent 3: Contract Judge

• Function: Evaluates a target code sample against the generated Gherkin specification.
• Mechanism: Performs a strict compliance check. Instead of asking "Is this code vulnerable?", it asks "Does this code satisfy the specific behavioral contract defined in the Gherkin spec?"
• Advantage: This transforms the task into a deterministic verification problem, allowing smaller models to perform effectively if the contract is clear.

3. Key Contributions

1. Phoenix Framework: A novel, training-free multi-agent architecture that achieves state-of-the-art results using fully open-source models (7–14B parameters) that are up to 48× smaller than previous SOTA models (e.g., DeepSeek-V3 671B).
2. Behavioral Contract Synthesis: The introduction of Gherkin specifications as a structured intermediate representation. This bridges the gap between code analysis and binary classification, resolving semantic ambiguity by defining security relative to specific contracts.
3. Comprehensive Ablation: Extensive experiments across 25 configurations and 5 model families proving that:
   • Gherkin specifications are the primary performance driver (+0.09 to +0.35 F1 improvement).
   • Code-specialized models (e.g., Qwen2.5-Coder-14B) are best for generating specifications (Agent 2).
   • Balanced reasoning models (e.g., Qwen3.5-9B) are best for judging compliance (Agent 3).

4. Experimental Results

Evaluated on the PrimeVul Paired Test Set (427 valid pairs, 859 samples):

• Performance Metrics:
  • F1 Score: 0.825 (vs. RASM-Vul's 0.668 and VulTrial's 0.563).
  • Pair-Correct (P-C): 64.4% (vs. RASM-Vul's 21.4% and VulTrial's 18.6%).
  • Note: P-C measures the ability to correctly classify both the vulnerable and patched versions of a pair simultaneously, a much stricter metric.
• Efficiency: Achieves these results using 7–14B parameter models without fine-tuning, whereas competitors often rely on 671B models or proprietary APIs (GPT-4o).
• Ablation Insights:
  • The "Blind" baseline (sliced code only, no Gherkin) achieved P-C of 19.9%, matching VulTrial's performance.
  • Adding the Gherkin specification boosted P-C to 64.4% (a 3.2× improvement), proving the gain comes from problem restructuring, not just model capacity.
• Error Analysis:
  • False Positives (FP): 18% of Phoenix's "False Positives" were actually genuine security concerns missed by the original developer patch (e.g., the patch fixed one issue but introduced a buffer overflow, or left a TODO comment acknowledging a known gap).
  • False Negatives (FN): Dominated by incomplete specifications or surface-level boundary checks (e.g., failing to detect that an unsigned integer check is logically vacuous).

5. Significance and Implications

• Paradigm Shift: The paper argues that security is a relative property defined against specific behavioral contracts, not an absolute syntactic feature. Phoenix validates this by treating detection as a contract verification problem.
• Beyond "Bigger Models": The results demonstrate that structured reasoning scaffolding (Gherkin specs) is more effective than simply scaling model parameters or relying on massive retrieval databases.
• Practical Application: The framework suggests a future where vulnerability detection integrates with Behavior-Driven Development (BDD). Instead of training models on massive datasets, systems can verify code against human-authored or reverse-engineered security contracts.
• Quality of "False Positives": The finding that Phoenix often flags "fixed" code as vulnerable (when the fix was incomplete or introduced new issues) highlights the system's potential as a rigorous security auditor rather than just a classifier, capable of identifying residual risks that human developers overlook.

In conclusion, Phoenix establishes that the bottleneck in vulnerability detection is not model intelligence but problem formulation. By decomposing the task into semantic slicing, contract synthesis, and strict compliance checking, it achieves superior accuracy with significantly smaller, open-source models.