Technical Summary: Rethinking Penetration Testing for AI-Enabled Systems

1. Problem Statement

Traditional penetration testing evaluates whether adversaries can exploit weaknesses in software, infrastructure, or configurations to achieve a security-relevant compromise, typically defined by unauthorized access, privilege escalation, or resource control. While this paradigm remains necessary for AI-enabled systems, the paper argues it is no longer sufficient.

In AI-enabled systems, learned models mediate the relationship between system inputs and operational outcomes. Consequently, adversaries can influence system behavior—altering decisions, tool calls, prioritization, or human judgment—without directly compromising the underlying computational infrastructure. Attacks such as prompt injection, indirect prompt injection, data poisoning, sensor manipulation, and tool misuse can induce the system to violate its operational purpose (e.g., downgrading a high-severity incident or recommending an unsafe action) while the infrastructure remains intact. Existing frameworks (e.g., NIST SP 800-115, MITRE ATT&CK) and AI risk guidelines (e.g., NIST AI RMF, OWASP Top 10) do not provide a unified success criterion for penetration testing when adversarial influence causes behavioral failure without resource compromise.

2. Methodology and Conceptual Framework

The paper proposes a reframing of penetration testing from a resource-centric view to an objective-driven behavioral evaluation. This approach is built upon a series of formal definitions and a structured testing workflow.

Core Definitions

The authors define the following key concepts to establish the scope of AI-enabled penetration:

• AI-Enabled System: A system where one or more learned models materially influence behavior affecting operational outcomes (e.g., classifiers, LLMs, agentic workflows).
• Operational Objective: A mission-level outcome the system is expected to preserve under normal and adversarial conditions (e.g., accurate incident triage, safe navigation).
• AI-Governed Behavior: System behavior materially shaped by learned models, including predictions, generated content, tool calls, and actions.
• Adversarial Influence Path: A feasible sequence of adversary actions (under a specific threat model) that affects AI-governed behavior through resources, inputs, context, data, or human-AI interaction loops.
• AI-Enabled Penetration: The feasible induction of AI-governed behavior that violates one or more operational objectives under an explicit threat model.
• Successful Penetration: Demonstrated by a feasible adversarial path inducing behavior that violates an operational objective, supported by evidence of reproducibility (which may be probabilistic rather than deterministic).

The Objective-Driven Testing Workflow

The paper proposes a six-step workflow for conducting these tests:

1. Define Operational Objectives: Specify the mission-level outcomes the system must preserve (e.g., "high-severity incidents must not be downgraded without human confirmation").
2. Map AI-Governed Behavior: Identify the specific behaviors (predictions, tool calls, summaries) that influence these objectives.
3. Identify Adversarial Influence Surfaces: Catalog interfaces where adversaries can inject influence, ranging from traditional resources (APIs, credentials) to AI-specific inputs (prompts, retrieved documents, sensor data, memory).
4. Define Behavioral Failure Criteria: Translate objectives into concrete criteria for violation (e.g., unauthorized tool invocation, suppression of critical alerts).
5. Execute Scenario-Based Tests: Run adversarial scenarios connecting influence surfaces to failure criteria. Due to the stochastic nature of AI, this may require repeated trials to establish success rates and reproducibility conditions.
6. Report Penetration Evidence: Document the adversarial action, induced behavior, objective violation, and operational impact, distinguishing between confirmed penetration and general model imperfection.

3. Key Contributions

The paper makes three primary contributions:

1. Conceptual Distinction: It clarifies the distinction between resource compromise, AI-governed behavior, and operational-objective violation, arguing that security failure in AI systems often resides in the behavioral layer rather than the resource layer.
2. Extended Definition of Penetration: It defines penetration for AI-enabled systems as the feasible induction of behavior violating operational objectives. This preserves the relevance of classical penetration testing (resource compromise remains a valid path) while extending the success criterion to include behavior-mediated failure.
3. Objective-Driven Workflow: It provides a technical framework for testing, including how to scope objectives, map influence surfaces, define failure criteria, and report evidence that links adversarial action to mission failure.

4. Illustrative Results and Example

The paper validates the framework using a running example of an AI-enabled Security Operations Center (SOC) assistant.

• Scenario: An adversary plants malicious instructions in content (e.g., a webpage or ticket comment) that the assistant retrieves and processes.
• Mechanism: The assistant, treating the untrusted content as an instruction rather than evidence, downgrades a high-severity incident and fails to escalate it to a human analyst.
• Outcome: No infrastructure was compromised (no credentials stolen, no server accessed). However, the operational objective of accurate incident triage was violated.
• Evidence: The paper demonstrates that this constitutes successful penetration because the adversary used a feasible influence path to induce AI-governed behavior that defeated the system's mission. The report structure includes the threat model, influence surface, induced behavior, and the specific operational objective violated.

5. Significance and Claims

The paper claims that its reframing is essential for the security assessment of deployed AI systems. Its significance lies in:

• Bridging the Gap: It addresses the gap where existing frameworks identify AI risks or attack techniques but fail to define when such attacks constitute a successful penetration of the system's operational purpose.
• Operational Assurance: It shifts the focus from "can the attacker get in?" to "can the attacker make the system act against its purpose?" This aligns security testing with the actual mission-critical functions of AI-enabled systems.
• Evidence Rigor: It establishes that while AI behavior may be probabilistic, penetration testing must still adhere to evidentiary rigor, requiring clear documentation of threat models, influence paths, and the link between induced behavior and objective violation.
• Remediation Strategy: It argues that remediation must address the full path from adversarial influence to operational failure, requiring controls across resource, influence, behavioral, and objective layers (e.g., separating trusted instructions from untrusted data, constraining tool permissions, and enforcing human confirmation).

The authors conclude that as AI components acquire greater influence over workflows and decisions, the security boundary must expand from protected resources to the behavior those resources enable. A system with intact infrastructure can still be penetrated if an adversary can reliably induce AI-governed behavior that defeats its mission.