Routing Ceilings Are Domain-Independent: Structural Prior Injection in Code Security Vulnerability Detection

Technical Summary: Routing Ceilings Are Domain-Independent

Problem Statement
Large Language Models (LLMs) exhibit a documented gap between their latent capabilities and their consistent activation, particularly when facing distributional shifts. Previous work by Cázares [1] introduced the router hypothesis, positing that LLMs solve problems by routing inputs to cached structural patterns rather than deriving answers compositionally. While injecting "structural priors" (e.g., cheatsheets) dramatically improves performance on in-distribution (ID) data, it causes a catastrophic collapse in out-of-distribution (OOD) settings, often performing worse than zero-shot baselines.

This paper investigates whether this phenomenon is domain-specific or a fundamental property of LLM reasoning. It asks if the "routing ceiling" and the associated distribution-shift collapse generalize from formal mathematical reasoning (the domain of the original study) to source-code security vulnerability detection, a high-stakes domain where synthetic benchmarks often fail to predict real-world performance.

Methodology
The authors replicate the experimental design of Cázares [1] within the BitCoder framework, evaluating three frontier LLMs (GPT-OSS-120B, Llama-3.3-70B, and Gemma-4-31B) across two distinct data distributions:

1. Synthetic Corpus (In-Distribution): A dataset of 348 labeled pairs (278 training, 70 held-out) covering three vulnerability categories:
   • Syntactic: CWE-798 (Hardcoded Credentials).
   • Contextual: CWE-284 (Improper Access Control/IDOR).
   • Semantic: N+1 Query Pattern (a non-CWE anti-pattern requiring runtime behavior understanding).
2. Real-World Corpus (Out-of-Distribution): The VUDENC benchmark, consisting of Python code fragments from real CVE-fixing commits, specifically targeting SQL Injection (CWE-89) and Path Traversal (CWE-22).

Experimental Conditions:

• Zero-shot: Standard prompt with task description.
• Cheatsheet v1: Prompts augmented with manually authored structural priors (canonical detection routes and signatures) derived from the synthetic training distribution.
• Cheatsheet v2: A revised cheatsheet for real-world data, calibrated using error patterns (false positives/negatives) identified in an initial VUDENC evaluation.

Key Contributions and Results

1. Replication of the Router Hypothesis in Code Security:
   The study confirms that structural priors saturate in-distribution performance. On synthetic data, cheatsheets lifted semantic-vulnerability (N+1) recall from 20.0% to 100.0% across all three models, with F1 scores reaching near-ceiling levels (up to 100%). This suggests the models possess the underlying capability but fail to activate it without explicit routing guidance.

2. Semantic Complexity Gradient:
   Zero-shot performance degrades predictably as semantic complexity increases. For GPT-OSS-120B, F1 scores followed the gradient: Syntactic (90.9%) > Contextual (85.7%) > Semantic (28.6%). This indicates that without structural priors, models struggle to route complex semantic reasoning tasks.

3. Amplified Distribution-Shift Collapse:
   The most critical finding is that the same cheatsheets that maximize synthetic performance cause a severe collapse on real-world CVE data.

   • Magnitude: For GPT-OSS-120B on CWE-89, the F1 score dropped from 100% (synthetic) to 48.9% (real), a −51.1 percentage point decline. This collapse is more than twice as severe as the −23.75 pp decline observed in the original mathematical domain study.
   • Comparison: The cheatsheet-augmented model performed significantly worse than the zero-shot baseline on real data (48.9% vs. 61.5%), confirming that the injected priors misalign with real-world patterns, systematically directing the model toward incorrect solution paths.

4. Failure of Iterative Recalibration:
   The authors tested whether refining the cheatsheet based on real-world error patterns (Cheatsheet v2) could mitigate the collapse. Instead, the v2 cheatsheet underperformed v1 (dropping to 41.7% F1 for CWE-89). This mirrors the "AN45c vs. AN38" finding from [1], suggesting that over-engineering priors for local splits exacerbates the structural mismatch with the target distribution.

5. Zero-Shot Misrouting:
   Under zero-shot conditions, models frequently detected vulnerabilities but assigned incorrect CWE labels (misrouting). While cheatsheets eliminated these misroutes, the authors note this is likely due to the cheatsheet directly supplying the category label rather than correcting the model's internal routing mechanism.

Significance and Claims
The paper argues that the cross-distribution trade-off surface is a domain-independent property of structural prior injection in LLMs. The findings suggest that:

• Prompt Engineering Limits: Relying on prompt engineering (checklists, few-shot exemplars, or structural routing) to achieve high benchmark scores may lead to silent degradation in production environments. The degradation is not a tuning problem solvable by better prompts but a structural consequence of distributional mismatch.
• Structural Nature of Collapse: The collapse is not a calibration artifact but a fundamental behavior where models route to cached patterns that do not generalize.
• Proposed Remedy: The authors argue that distribution-aware training (fine-tuning on hybrid synthetic and real datasets) is a more promising path than prompt calibration. By embedding distributional knowledge into the model's weights rather than the prompt, the internal routing prior may better align with the target domain.

The study concludes that the router hypothesis generalizes from formal mathematics to code security, providing evidence that LLMs activate learned sub-skills via structural routing mechanisms that are highly sensitive to distribution shifts.