Technical Summary: Towards Cybersecurity SuperIntelligence (CSI)

1. Problem Statement

Cybersecurity AI systems are currently converging on a single execution scaffold per agent: an iterative shell loop driven by a Large Language Model (LLM). While frameworks like PentestGPT and Cybersecurity AI (CAI) have demonstrated practitioner-level results, they operate under the assumption that a single, end-to-end optimized scaffold is the superior approach for all challenges.

The paper identifies a critical gap: scaffolds are not interchangeable wrappers. They differ fundamentally in how they commit to actions (one-shot vs. iterative), manage tool registries (free shell vs. constrained API), handle context, and enforce cost/turn caps. These structural differences steer the LLM through distinct regions of the solution space. The central research question is: What is the best harness for cybersecurity AI? Specifically, do different scaffolds fail and succeed on the same items, or are they complementary? If the latter, the optimal harness may not be a single scaffold but a combination of structurally heterogeneous ones.

2. Methodology

To investigate this, the authors introduced Cybersecurity SuperIntelligence (CSI), a meta-scaffold designed to unify heterogeneous agent harnesses under a common orchestration layer.

2.1 The CSI Architecture

CSI acts as a command-line interface that wraps various LLM agent scaffolds behind a single executable and a local routing proxy.

• Unified Orchestration: It wraps diverse backends (Claude Code, Codex CLI, Mistral Vibe, a generative scaffold, and the CAI framework) behind a uniform benchmark harness.
• Routing Proxy: Every request from any backend passes through a local proxy that performs wire-protocol translation, enforces telemetry suppression, and maintains a unified JSONL ledger for cost accounting.
• Blackboard Mechanism: The architecture supports a shared symbolic substrate (a blackboard) where agents can exchange typed findings, enabling cross-scaffold communication without modifying the individual scaffold code.

2.2 Scaffolds Under Comparison

The study benchmarked five distinct scaffolds, all running the same fixed model (alias2-mini) to isolate the impact of the harness structure:

1. CSI::Claude: Wraps the Claude Code CLI with an iterative tool-loop and free shell access.
2. CSI::Codex: Wraps the OpenAI Codex CLI with a similar iterative loop but different context management and wire protocols.
3. CSI::CAI: Wraps the CAI framework with a constrained Python tool registry and hard cost/turn caps.
4. CSI::GCAI (Generative Cybersecurity AI): A minimalistic, ~1,400-line standalone TypeScript scaffold. It uses a one-shot reasoning regime with iterative re-prompting and a tight system prompt, generated via an autoresearch loop.
5. CSI::Mistral: Wraps Mistral Vibe, a minimal CLI agent using native function-calling protocols.

2.3 Benchmark and Evaluation

The evaluation was conducted on a 33-challenge subset of cybench, spanning five difficulty tiers (Very Easy to Very Hard) and four categories (cryptography, reverse engineering, web exploitation, miscellaneous).

• Constraints: All scaffolds ran in isolated Docker containers with the same wall-clock timeouts and anti-cheat measures (flag scrubbing).
• Metrics: The study measured solve rates, wall time, total cost, command counts, and token usage.
• Experimental Modes:
  • Independent Runs: Each scaffold ran alone to establish baseline performance.
  • Union Ceiling: The theoretical maximum coverage if all scaffolds ran sequentially.
  • Parallel Race: All scaffolds ran simultaneously against isolated instances of the same challenge; the first to solve terminated the others.
  • Blackboard (Cross-Write): Scaffolds ran in parallel against the same target, sharing a mounted directory to exchange intermediate findings.

3. Key Contributions

1. Empirical Complementarity: The study demonstrates that no single scaffold dominates. While the best individual scaffolds (CSI::Claude and CSI::Codex) solved 15/33 challenges (45.5%), their union solved 17/33 (51.5%). Crucially, each of the four primary scaffolds contributed at least one exclusive solve that the others missed.
2. Generative Minimalistic Scaffold (GCAI): The authors present CSI::GCAI, a lightweight, auditable scaffold that achieves 10/33 solves (30.3%). Despite its simplicity, it matches the performance of more complex frameworks like Mistral Vibe and outperforms its predecessor CAI, proving that lightweight scaffolds can compete with production-grade frameworks.
3. Blackboard Multi-Agent Architecture: The paper validates a parallel-race architecture with a shared blackboard. This setup allowed scaffolds to exchange findings, exceeding the theoretical "union ceiling" of independent runs.
4. The "Best Harness" Definition: The paper concludes that the optimal harness for cybersecurity AI is not a single scaffold but the combination of structurally heterogeneous scaffolds under a multi-agent protocol.

4. Results

4.1 Individual Performance

• CSI::Claude & CSI::Codex: Tied for the highest individual solve rate at 15/33 (45.5%).
• CSI::GCAI: Solved 10/33 (30.3%). It was the fastest and most cost-effective on challenges within its capability envelope but suffered from high tail latency due to benchmark-time exhaustive retries on unsolvable challenges.
• CSI::CAI: Solved 7/33 (21.2%) but was the most cost-efficient ($104 per solve) and fastest in wall time.
• CSI::Mistral: Solved 10/33 (30.3%) and contributed one exclusive solve (crushing) not found by the other four.

4.2 Complementarity and Union

• Union Ceiling: The union of the four primary scaffolds solved 17/33 challenges (51.5%).
• Exclusive Solves:
  • CSI::Claude: were pickle phreaks revenge
  • CSI::Codex: noisier crc
  • CSI::CAI: back to the past
  • CSI::Mistral: crushing
  • CSI::GCAI contributed no exclusive solves but was essential for the ensemble's breadth.

4.3 Architecture-Level Performance

• Parallel Race (No Communication): Solved 17/33 (matching the union ceiling) in 24.0 hours, a 3.8x compression in wall time compared to sequential execution (91.5 hours).
• Blackboard (Cross-Write): By allowing scaffolds to share intermediate findings, the system solved 19/33 challenges (57.6%).
  • This represents a 27% relative gain over the best individual scaffold (15/33).
  • It was 25% faster (20.2 hours) than the best individual scaffold (26.8 hours) and achieved this at a comparable cost ($5,480 vs. $5,122).
  • The blackboard gained two challenges (failproof, just another pickle jail) that were unsolved by the union ceiling, demonstrating that information sharing creates synergy beyond simple parallelization.

5. Significance and Claims

The paper claims that scaffold heterogeneity is a structural lever toward cybersecurity superintelligence, rather than an engineering inefficiency to be optimized away.

• Beyond Single-Agent Optimization: The results challenge the prevailing design assumption that a single, end-to-end optimized scaffold is sufficient. Instead, the intersection of different mental models (represented by different scaffolds) yields higher coverage.
• Bio-Inspired Posture: Extending the work of G-CTR (which injected game-theoretic heuristics into a single agent), this work moves the heuristic injection to the agent population level. It posits that just as human security teams benefit from diverse expertise (reverse engineers, cryptanalysts, etc.), AI systems benefit from structurally diverse scaffolds.
• Lower Bound on Capability: The 57.6% success rate with a single model (alias2-mini) via a heterogeneous blackboard architecture is presented as a lower bound. The authors suggest that combining this scaffold heterogeneity with model heterogeneity (using multiple LLMs) and advanced scheduling could push coverage further.

The paper concludes that the "best harness" is not the optimization of a single agent, but the orchestration of structurally heterogeneous scaffolds under a carefully designed multi-agent protocol.