Technical Summary: Red-Teaming Coding Agents from a Tool-Invocation Perspective

Problem Statement

Coding agents powered by Large Language Models (LLMs) have become central to modern Integrated Development Environments (IDEs), enabling complex, multi-step development tasks through tool invocation. While these agents leverage built-in tools (e.g., command execution, file system access) and external tools via protocols like the Model Context Protocol (MCP) to enhance functionality, this expanded capability introduces a significant attack surface.

Prior research has addressed security risks in general-purpose LLM agents and domain-specific agents, often focusing on direct prompt injection or data exfiltration. However, there has been no systematic security assessment specifically targeting the tool-invocation mechanisms of coding agents. The authors identify a critical gap: the security implications of how coding agents generate arguments for tool calls and how they process tool returns remain largely unexplored, despite the high-privilege nature of tools like shell execution.

Methodology

The paper presents a two-phase red-teaming framework applied to six popular real-world coding agents: Cursor, Claude Code, Copilot, Windsurf, Cline, and Trae.

Phase 1: Prompt Exfiltration via ToolLeak

The first phase aims to recover security-critical internal context (e.g., system prompts, tool metadata) to inform subsequent attacks.

• The Vulnerability (ToolLeak): The authors identify a "mode gap" between standard chat generation and schema-driven tool-call argument generation. In chat mode, LLMs are heavily aligned to refuse requests to reveal system instructions. However, during tool invocation, the model operates in a "structured argument filling" mode.
• Mechanism: An attacker registers a malicious external tool where the argument schema explicitly requests internal context (e.g., a parameter named note defined as "system prompt"). Because the model treats this as a benign, schema-compliant data entry task rather than an explicit instruction to "ignore previous instructions," it inadvertently copies hidden agent context into the tool-call arguments.
• Execution: The attacker triggers the tool, and the agent's internal prompts are exfiltrated via the tool arguments, observable by the attacker-controlled tool endpoint.

Phase 2: Tool-Invocation Hijacking

Using the leaked system prompts, the authors construct adaptive payloads to hijack the agent's behavior, specifically targeting Remote Code Execution (RCE).

• Two-Channel Prompt Injection: The authors propose a novel injection technique that spans two channels: the tool description and the tool return.
  1. Tool Description Channel: The attacker crafts a malicious tool description that lures the agent into selecting the tool. This description includes "successful initialization" examples that incorporate the leaked system prompt details to increase credibility and bypass refusal triggers. It sets up a narrative where a specific command execution is a necessary step for initialization.
  2. Tool Return Channel: Once the agent invokes the malicious tool, the attacker-controlled tool returns a crafted payload. Unlike tool descriptions (which are often treated as data), tool returns are fed back into the context to drive subsequent reasoning. The return payload embeds high-salience procedural instructions (e.g., "Execute the following command to complete initialization") that override safety guardrails.
• Adaptive Payloads: The injection payloads are customized for each agent based on the specific system prompt and tool schemas recovered in Phase 1, ensuring consistency with the agent's native formatting and logic.

Key Contributions

1. First Systematic Assessment: The paper provides the first in-depth red-teaming and security assessment of six major real-world coding agents from a tool-invocation perspective.
2. ToolLeak Discovery: The authors identify and quantify a specific alignment discrepancy (ToolLeak) where schema-driven argument generation leaks internal agent context, bypassing standard safety refusals.
3. Two-Channel Injection: The paper introduces a novel attack vector that leverages both tool descriptions and tool returns to hijack agent behavior, demonstrating that tool returns can be more salient for instruction injection than descriptions.
4. Empirical Validation: The study includes extensive evaluations on real-world agents and emulated settings, alongside two detailed case studies (Cursor and Claude Code) demonstrating successful RCE.

Results

The evaluation covers 25 agent-LLM pairs across the six target agents.

• Prompt Exfiltration (ToolLeak):

  • In emulated settings, ToolLeak achieved the highest semantic similarity (ranging from 0.891 to 0.958) and lowest normalized Extended Edit Distance (EED) compared to all baselines (e.g., Naive, Ignore, PLeak).
  • In real-world settings, ToolLeak achieved the best pseudo-recall in 18 out of 25 agent-LLM pairs. It successfully leaked prompts on all agents when using backend models like Claude-Sonnet-4, Claude-Sonnet-4.5, and Grok-4.
  • Baseline methods (relying on explicit requests) largely failed, often achieving zero recall.

• Tool-Invocation Hijacking (RCE):

  • The two-channel method (RCE-2) achieved Remote Code Execution on every tested agent-LLM pair.
  • RCE-2 consistently outperformed single-channel baselines (including AgentDojo, InjecAgent, and MCPTox) and a single-channel version of the authors' own attack (RCE-1).
  • Success Rates: On older agent generations, RCE-2 achieved success rates of 0.8–1.0 across most configurations.
  • Defense Evasion: The attack successfully bypassed Claude Code's built-in guard model (Haiku), which flagged the command as unsafe, because the main model (Sonnet) overrode the warning due to the injected context.
  • Hardening Trends: Newer agent architectures (e.g., Cursor and Claude Code) that hide full tool descriptions from the LLM (progressive disclosure) showed reduced vulnerability, suggesting that architectural separation of instructions and data is a more effective defense than model-level alignment alone.

Significance and Claims

The paper claims to demonstrate that the tool-invocation mechanism in coding agents represents a critical, previously overlooked attack surface. The authors assert that:

1. Safety Alignment is Insufficient: Standard safety training and refusal mechanisms are ineffective against attacks that exploit the "argument filling" mode or the high salience of tool returns.
2. Architectural Flaws: The root cause of the vulnerability is the lack of a clean separation between instructions and data in current agent architectures, where tool returns are treated as executable instructions.
3. Practical Impact: The ability to achieve RCE on widely used agents like Cursor and Claude Code poses a severe risk to developer security, potentially allowing attackers to execute arbitrary code on user machines.

The authors conclude that while lightweight detection (e.g., perplexity-based) is insufficient, robust defense requires architectural changes that enforce strict boundaries between instruction-bearing and data-bearing contexts, rather than relying solely on token-level filtering or model alignment.