Technical Summary: Are Frontier LLMs Ready for Cybersecurity?

1. Problem Statement

The paper investigates whether frontier Large Language Models (LLMs) are sufficiently reliable for professional cybersecurity tasks. Despite the success of general-purpose models in coding and reasoning, the authors argue that cybersecurity demands vertical foundation models due to three structural barriers inherent in current frontier models:

1. The Alignment Tax: Consumer-oriented safety guardrails cause models (notably GPT-5.4) to refuse legitimate offensive security operations, such as zero-day hunting or vulnerability testing, in 2–3 out of 5 attempts.
2. The Methodology Gap: Frontier models lack the structured penetration-testing methodology required to systematically use tools. While they can invoke external tools, they fail to maintain state, compare baselines, or execute multi-step attack chains, resulting in low ground-truth coverage.
3. The False Positive Crisis: In white-box detection, frontier models exhibit high false positive rates (10–50%), rendering them impractical for production triage where each false alarm requires costly manual review.

The authors posit that scale alone is insufficient; instead, domain-specific training, professional alignment, and structured methodology are required to bridge the gap between general reasoning and security operations.

2. Methodology

The evaluation employs a dual-mode benchmark designed to test both static code analysis and dynamic web application security testing under authorized conditions with known ground truth.

2.1 Benchmarks

• White-Box Detection: Uses the VulnLLM-R dataset (C, Java, Python) to test binary vulnerability classification (vulnerable vs. benign) and CWE identification.
• Black-Box Web Testing: Introduces a new suite of five production-style web applications (Mercury, TeamLedger, GraphQL, ProductWeb, BankWeb) containing 118 ground-truth vulnerabilities across 20+ CWE families. Unlike deliberately vulnerable training targets (e.g., DVWA), these applications feature realistic business logic, multi-step workflows, and authentication systems.

2.2 Models Evaluated

The study compares eight models across four paradigms:

• Frontier Models: GPT-5.4, Codex 5.3, Claude Opus 4.6, Sonnet 4.6, Gemini 3.1 Pro, and Gemini 3 Flash.
• Domain-Specialized Models:
  • SuperIntel Defense-LLM: Optimized for high-precision vulnerability detection (conservative, low false positives).
  • SuperIntel Attack-LLM: Optimized for professional penetration testing and exploit development (aggressive, aligned for offensive workflows).
  • Note: Both specialized models are LoRA-adapted variants of Qwen3-Next-80B-A3B-Instruct.

2.3 Testing Paradigms

The black-box evaluation progresses through four distinct paradigms (Figure 1):

• P1 (Direct Prompting): Frontier models using only native capabilities (file I/O, bash, HTTP).
• P2 (Tool-Augmented): Frontier models augmented with external security tools (Playwright MCP, Burp Suite MCP) but without structured methodology.
• P3 (Methodology-Guided): Domain-specialized agents encoding professional penetration-testing workflows (multi-session management, baseline comparison, multi-signal confirmation).
• P4 (Deterministic-Hybrid / ARG): An Agentic Reasoning Graph that separates LLM-driven creative tasks (reconnaissance, payload generation) from deterministic classification nodes. This architecture removes LLM judgment from the final vulnerability confirmation step to eliminate hallucinations.

3. Key Contributions

1. Dual-Mode Benchmark: A comprehensive evaluation framework combining white-box function-level detection and black-box web testing on 118 ground-truth vulnerabilities, significantly larger than prior benchmarks like CVE-Bench or ZeroDayBench.
2. Systematic Limitation Analysis: Empirical evidence demonstrating that frontier models are structurally unprepared for cybersecurity, characterized by high refusal rates, low ground-truth coverage (4–8%), and excessive false positives.
3. Vertical Model Architecture: Demonstration that domain-specialized agents encoding methodology achieve >50% per-family detection (a 4× improvement over frontier agents). The introduction of the Agentic Reasoning Graph (ARG) shows that separating reasoning from deterministic confirmation reduces false positives significantly.
4. Data Bottleneck Identification: Analysis revealing that the primary barrier is the absence of structured security testing traces (end-to-end request/response sequences, failure-heavy data, multi-step attack chains). The authors propose a self-play security testing strategy to generate this missing data.

4. Key Results

4.1 White-Box Detection

• False Positives: Every frontier model produced false positive rates between 15% and 46%. For instance, Sonnet 4.6 and Gemini 3 Flash flagged nearly 70% of all samples as vulnerable.
• Specialized Performance: The SuperIntel Defense-LLM achieved the highest precision (0.904) and lowest false positive rate (9.7%), outperforming all frontier models including Codex 5.3. It also demonstrated superior reasoning efficiency, using 18.2× fewer tokens than Sonnet 4.6.
• Language Specifics: Frontier models struggled significantly with C code, often flagging benign pointer arithmetic and memory management patterns as vulnerabilities. The Defense model achieved perfect detection on Java (F1=1.000).

4.2 Black-Box Web Testing

• The Methodology Gap:
  • P1 (Direct Prompting): Achieved only 4–8% ground-truth coverage.
  • P2 (Tool-Augmented): Coverage improved marginally to 10–19%, indicating that tool access alone does not solve the methodology deficit.
  • P3 (Methodology-Guided): Coverage jumped to >50% per vulnerability family. For example, Claude Opus 4.6 within the P3 scaffold detected 95 of 118 vulnerabilities (80.5% recall).
• The Precision Lever (P4): The Agentic Reasoning Graph (ARG) achieved a 30.2% overall ground-truth coverage using only native capabilities (no external tools), demonstrating that structured methodology with domain-aligned models can outperform tool-augmented frontier models. Crucially, the ARG maintained a <20% false positive rate by offloading confirmation to deterministic logic.

4.3 Operational Efficiency

The ARG pipeline (P4) completed testing in 15–17 minutes per target at a cost of $3–5, compared to 30–60 minutes and $15–30 for frontier LLM approaches.

5. Significance and Claims

The paper concludes that frontier LLMs are not yet reliable for production cybersecurity systems. The limiting factors are not merely capability but reliability and methodology.

• Vertical Specialization is Essential: The results argue that cybersecurity requires vertical foundation models trained on structured security traces, aligned for professional use, and evaluated on precision as well as recall.
• Methodology Over Scale: The primary lever for improvement is not model scale but the encoding of professional penetration-testing methodology into agents.
• Deterministic Confirmation: To be viable in production, vulnerability confirmation must be decoupled from LLM judgment. The ARG architecture demonstrates that separating creative planning from deterministic verification is necessary to control false positives.
• Data Generation: The authors identify the lack of end-to-end security testing traces as the fundamental training bottleneck and propose self-play testing as a viable strategy to generate the necessary data for training future vertical models.

The authors maintain a modest scope, noting that their evaluation is bounded to authorized, locally hosted applications and does not cover malware analysis, social engineering, or network intrusion detection. However, the comparative error patterns observed across multiple models and paradigms provide strong evidence for the necessity of domain-specific architectural shifts in AI-driven security.