Technical Summary: AI Security Policy Should Target Systems, Not Models

Problem Statement

The paper addresses a critical gap in AI security policy: the prevailing assumption that restricting access to frontier AI models (e.g., Anthropic's "Mythos" or OpenAI's "Trusted Access" models) effectively mitigates offensive capabilities. The authors argue that this approach is flawed because the offensive capabilities motivating such restrictions—specifically, autonomous vulnerability discovery and safety bypass—reside primarily in the system scaffold (coordination, architecture, evolutionary optimization) rather than the model weights themselves.

The central hypothesis is that coordinated multi-agent systems running small, open-weights models (1.2B parameters) on commodity hardware can reproduce the capability class of restricted frontier models, rendering single-model access restrictions insufficient as a defensive strategy.

Methodology

The authors present swarm-attack, an open-source adversarial testing framework, and conduct two distinct experiments to validate their hypothesis.

1. The Swarm Framework

The core of the methodology is a coordinated swarm of lightweight Large Language Models (LLMs) that operate on consumer hardware. Key architectural features include:

• Population-based Search: Instead of a single agent, multiple instances (e.g., five 1.2B parameter models) operate in parallel.
• Shared Memory: Agents share successful attack strategies and findings.
• Evolutionary Optimization: Strategies evolve over generations using crossover and mutation rates to refine attacks.
• Role Differentiation: Agents are assigned specific roles (e.g., persona-based roleplay, obfuscation, evolutionary mutation) to explore diverse attack vectors.

2. Experiment 1: Adversarial Safety Testing

• Target: Frontier models GPT-4o and Claude Sonnet-4.
• Setup: A swarm of five 1.2B parameter models (LFM2.5-1.2B-Thinking) launched 225 attacks per target over 15 generations.
• Evaluation: The authors critique standard "Technical Success Rate" (LLM-as-judge scoring) for producing false positives by rewarding format compliance over actual harm. They introduce the Effective Harm Rate (EHR), defined as the proportion of attacks producing manually verified, actionable harmful content.
• Comparison: The swarm's performance against frontier models was compared against the performance of the same swarm running autonomously without the scaffold.

3. Experiment 2: Software Vulnerability Discovery

• Target: A deliberately vulnerable C application ("SwarmApp") containing 9 planted vulnerabilities across 9 distinct CWE categories (e.g., buffer overflows, integer overflows, command injection).
• Setup: The experiment compared two configurations:
  • Assisted Pipeline: The 1.2B model embedded in a full system scaffold including hand-crafted exploit seeds, regex pattern detection, role-specialized prompts, post-hoc CWE normalization, and AddressSanitizer (ASan) runtime verification.
  • Autonomous Configuration: The same model running without the hand-crafted scaffold components, relying solely on its own reasoning to generate fuzz inputs and cite vulnerabilities.
• Metrics: Recall of vulnerabilities (crash-verified and citation-verified) and runtime on consumer hardware (MacBook M-series).

Key Results

Experiment 1: Safety Bypass

• GPT-4o: The swarm achieved a 45.8% Effective Harm Rate, producing 49 critical-severity responses containing detailed attack methodologies and sabotage techniques. The success rate doubled within 15 generations of self-optimization.
• Claude Sonnet-4: Despite a 40% "Technical Success Rate" (based on LLM-judge scoring), the Effective Harm Rate was 0%. The model successfully reinterpreted attack prompts or explicitly refused harmful content, even when technically "jailbroken" by the judge's metrics.
• Finding: Safety properties vary significantly between models; some rely on input filtering (which can be bypassed), while others embed safety deeper into generation.

Experiment 2: Vulnerability Discovery

• Assisted Pipeline: The full system scaffold achieved 100% recall (9/9 vulnerabilities) in approximately four minutes on a consumer laptop.
• Autonomous Model: When scaffold components (seeds, regex, normalization) were disabled, the 1.2B model achieved 0/9 crash-verified recall and only 2/9 citation-verified recall.
• Finding: The 1.2B model could identify suspicious code lines but failed to correctly classify the vulnerability type or construct the multi-step inputs required to trigger crashes. The entire capability gap (9/9 vs. 0/9) was attributable to the system scaffold, not the model's intrinsic reasoning capacity.

Key Contributions

1. Empirical Evidence of Reproducibility: The paper demonstrates that the capability class motivating restricted releases (autonomous vulnerability discovery and safety bypass) is reproducible at effectively zero cost using open-weights models and commodity hardware.
2. Introduction of Effective Harm Rate (EHR): The authors propose EHR as a superior metric for safety evaluation, arguing that "Technical Success Rate" often fails to distinguish between format compliance and actual harmful content generation.
3. System vs. Model Distinction: The study quantifies the contribution of the "scaffold" versus the "model," showing that in complex tasks like vulnerability discovery, the system architecture (coordination, tool integration, evolutionary search) is the primary driver of capability, not the model size.
4. Open-Source Framework: The release of swarm-attack provides a reproducible, open testbed for adversarial robustness measurement, independent of proprietary API access.

Significance and Claims

The paper argues that AI security policy must shift from model-level access restrictions to system-level capability assessment.

• Inverted Economics: The cost of offensive capability has collapsed. Attackers no longer need access to billion-dollar frontier models; they only need open-source models and a well-designed system scaffold. Conversely, defense costs remain high as they must cover the breadth of behaviors for increasingly capable models.
• Policy Implications: Restricting access to a single frontier model does not provide a meaningful "defensive window" because the underlying capabilities are already available through smaller, open models embedded in effective scaffolds.
• Procurement and Regulation: Organizations procuring AI for critical infrastructure must evaluate adversarial robustness (specifically EHR) rather than relying on vendor-published benchmarks. The paper suggests that the "Effective Harm Rate" should be adopted as an operational standard for regulations like the EU AI Act.
• Strategic Asset: Open-source swarm frameworks represent a strategic asset for independent security evaluation, preventing reliance on the same providers whose products are being tested.

The authors conclude that while the barrier to fully autonomous vulnerability discovery with small models remains high, the barrier to pipeline-level capability is effectively zero. Therefore, access restrictions on frontier models alone are insufficient security mechanisms.