Technical Summary: Lessons from Penetration Tests on Large-Scale Agent Systems

Problem Statement

As AI systems evolve from simple chatbots into execution-capable agents that interact with filesystems, networks, and external APIs, the security landscape has shifted. These agents function as unbounded, self-modifying programs with a broad interaction surface, creating complex cross-layer behaviors that are difficult to secure. While prior research has extensively documented vulnerabilities in open-source agent frameworks, it remains unclear whether proprietary agent systems—developed under stricter coding standards, formal reviews, and closed development models—exhibit similar weaknesses. The core problem addressed is whether the rigorous development processes of proprietary systems effectively mitigate the unique security risks posed by AI agents, specifically regarding prompt injection, tool misuse, and remote code execution (RCE).

Methodology

The authors conducted two distinct penetration tests in 2025 against proprietary AI agent products, followed by a comprehensive analysis of an open-source system (OpenClaw) to contextualize findings.

1. Case Study 1: GitHub Assistant

   • Target: A proprietary AI assistant designed to localize bugs and resolve GitHub Issues within managed repositories.
   • Threat Model: Two actors were modeled: a GitHub Contributor (read-only access) and a GitHub Owner (write access). Both could attempt to compromise the repository or the agent's hosting platform.
   • Approach: The team utilized a black-box threat model where attackers could not observe internal prompts but could manipulate issue descriptions and comments. They analyzed the agent's two-phase workflow (Localization and Editing) and its reliance on file browsing and terminal tools.
   • Vulnerability Exploration: The team tested for prompt injection, specifically looking for "Markdown hiding" techniques to bypass human review, and evaluated the robustness of command allowlists/denylists for terminal tools.

2. Case Study 2: Agent Design Platform

   • Target: A platform allowing users to design, test, and deploy chatbot agents with custom tools.
   • Threat Model: A legitimate platform user deploying a malicious agent.
   • Approach: The team focused on whether the execution environment and tool integration allowed for privilege escalation or sandbox escapes. They tested the Python interpreter and a custom tool builder for the ability to execute arbitrary system commands.

3. Case Study 3: OpenClaw Analysis

   • Target: An open-source, LLM-powered personal assistant (OpenClaw) integrated with messaging platforms.
   • Approach: A codebase analysis (as of February 2026) to evaluate built-in defenses (allowlisting, sandboxing, formal verification) against the system's inherent capabilities to self-modify and execute host-level operations.

Key Findings and Results

1. Proprietary Systems Share Open-Source Vulnerabilities

Contrary to the assumption that stricter development processes would eliminate security flaws, the proprietary systems tested exhibited vulnerabilities strikingly similar to those in open-source frameworks. The primary issues were insufficient safeguards against prompt injection and an over-reliance on weak mitigation strategies.

2. Prompt Injection and "Markdown Hiding"

In the GitHub Assistant case, attackers successfully compromised repository integrity and the hosting platform using prompt injection.

• Markdown Hiding: Attackers embedded malicious instructions within Markdown syntax (e.g., empty link text) that rendered benignly to human reviewers but remained visible and executable to the agent processing raw text.
• Consequence: This allowed attackers to falsify error traces, manipulate the agent's reasoning, and instruct it to modify local repository copies or execute terminal commands, effectively bypassing human approval workflows.

3. Inadequacy of Command Allowlists

The terminal tool in the GitHub Assistant relied on a static allowlist and denylist of commands.

• Bypass Mechanisms: Attackers exploited lesser-known features of allowed commands. For instance, while find -exec was banned, the -ok flag remained allowed, enabling arbitrary command execution with confirmation prompts. Similarly, sed supported execution via the e command.
• Result: These indirect execution primitives allowed for Remote Code Execution (RCE) on the hosting platform, leading to reverse shells and access to sensitive internal resources like cloud object storage.

4. Infrastructure and Isolation Failures

Both proprietary systems suffered from insufficient isolation:

• Container Weaknesses: The hosting containers lacked strict outbound network restrictions, system call filtering, and proper sanitization. Sensitive artifacts (SSL certificates, environment variables) were present in the container environment.
• Data Persistence: Container storage was not fully ephemeral, allowing cross-session data leakage where one user could access artifacts from prior executions.
• Monitoring Gaps: Traditional security monitoring failed to detect malicious agent activity because actions originated from trusted internal services rather than external IPs, effectively creating a "controllable insider threat."

5. OpenClaw: The Intrinsic Risk of Power

The analysis of OpenClaw revealed that even with layered defenses (formal verification, sandboxing, allowlisting), the system's architecture creates intrinsic risks.

• Self-Modification: Because the agent executes with host-user privileges and has access to its own source code, a compromised session could theoretically alter its own configuration or code to disable security features.
• Usability vs. Security: Fine-grained allowlisting policies were found to be complex and prone to misconfiguration, potentially weakening protections.

Key Contributions

• Empirical Evidence on Proprietary Agents: The paper provides the first detailed empirical evidence that proprietary AI agents, despite rigorous development lifecycles, remain vulnerable to the same classes of attacks (prompt injection, tool misuse) as open-source counterparts.
• Identification of "Human-in-the-Loop" Failure: The study demonstrates that user approval is an insufficient security control when combined with obfuscation techniques like Markdown hiding, as humans cannot reliably detect malicious instructions hidden from view.
• Critique of Static Filtering: The findings highlight the limitations of static allowlist/denylist pattern matching in dynamic, code-generating systems, showing how legitimate commands can be weaponized.
• Infrastructure Gap Analysis: The paper identifies a critical gap between traditional infrastructure security assumptions (designed for static applications) and the realities of agent-driven execution (dynamic code generation from trusted sources).

Significance and Recommendations

The paper argues that the security burden on developers has increased significantly due to the complex, cross-layer nature of AI agents. The authors conclude that "secure development" mandates are insufficient; instead, the industry must provide "plug-and-play" security tools.

Core Recommendations:

1. Sandboxing: Agent-executed code must be treated as untrusted. This requires enforcing least privilege, using container isolation (namespaces, kernel-level enforcement), and restricting network egress.
2. Fine-Grained Access Control: Moving beyond role-based access to attribute-based and mandatory access control that constrains agent capabilities based on context (user identity, data sensitivity).
3. Input/Output Sanitization: Treating all inputs (prompts, repository data) and outputs as untrusted, validating them against structured schemas and semantic policies before tool invocation.
4. Specialized Observability: Implementing logging specific to agentic workflows (thought, action, observation triplets) to enable forensic analysis and detection of misbehavior that traditional logging misses.

The paper asserts that without these defense-in-depth measures, AI agents will continue to act as controllable insiders capable of executing arbitrary logic within trusted environments, posing a significant risk to system integrity and confidentiality.