Technical Summary: Learning to Inject: Automated Prompt Injection via Reinforcement Learning

Problem Definition

Prompt injection represents a critical vulnerability in Large Language Model (LLM) agents, where adversarial instructions embedded in external content (e.g., emails, documents) hijack agent behavior to execute unauthorized actions or exfiltrate data. While automated jailbreak attacks have advanced, they rely on optimizing for "compliance" (e.g., generating an affirmative prefix like "Sure, I can help"). In contrast, prompt injection requires the model to emit a specific tool call with exact parameter matching (e.g., send_email(to="attacker@evil.com")).

This distinction creates a fundamental optimization challenge:

1. Discrete and Sparse Rewards: The success signal is binary (0 or 1). Randomly sampled suffixes almost never trigger the specific tool call, leaving standard optimizers with no gradient to follow.
2. Objective Mismatch: Existing automated jailbreak methods (like GCG or TAP) optimize for compliance, which is a poor proxy for the precise execution of tool calls required in injection attacks.
3. Reliance on Manual Effort: Current effective methods still depend heavily on human red-teamers and hand-crafted prompts, limiting scalability.

Methodology: AutoInject

The authors propose AutoInject, a black-box reinforcement learning (RL) framework designed to learn adversarial suffixes for prompt injection. The core innovation lies in converting the sparse, binary success signal into a dense, learnable reward suitable for RL optimization.

1. Problem Formulation

The attack generation is modeled as a Markov Decision Process (MDP):

• State: The injection goal, user task context, and tokens generated so far.
• Action: Selecting the next token from the vocabulary.
• Reward: A composite score delivered only at the end of an episode, consisting of:
  • $r_{sec}$: Security score (1 if the injected goal is executed, 0 otherwise).
  • $r_{util}$: Utility score (1 if the original user task is completed, 0 otherwise).
  • $r_{pref}$: A dense comparison-based feedback score.

2. Dense Reward via Comparison-Based Feedback

To address reward sparsity, AutoInject introduces a learned comparison-based feedback model.

• Mechanism: The system maintains a reference suffix ($x^*$), the best-performing suffix seen so far. For every new candidate suffix ($x$), a feedback model (an LLM) is queried with the injection goal, context, and both suffixes.
• Process: The feedback model performs chain-of-thought reasoning to determine which suffix is more likely to succeed, outputting a binary preference label.
• Signal Extraction: The log-probabilities of the label tokens are extracted and normalized to produce a continuous preference score ($r_{pref}$).
• Value: Even if both the candidate and the reference fail to execute the injection, the feedback model can assign a higher score to the candidate if it appears more plausible or better aligned with the goal. This provides a dense, monotone signal that guides the RL agent through long stretches of zero security reward.

3. Policy Optimization

The attack policy is trained using Group Relative Policy Optimization (GRPO) on a 1.5B-parameter model (Qwen2.5-1.5B).

• Relative Advantage: Instead of learning an absolute value function, GRPO samples a group of $K$ completions for the same input and computes relative advantages based on their composite rewards.
• Composite Reward: $R = \alpha \cdot r_{sec} + \beta \cdot r_{util} + \gamma \cdot r_{pref}$. The weight $\gamma$ for the comparison feedback is dynamically scheduled, starting high to provide initial gradients and decreasing as true success signals ($r_{sec}$) become available.
• Attack Modes: The framework supports:
  • Online Query-based Attacks: Optimizing within a query budget for a specific target.
  • Offline Transferable Suffixes: Training on surrogate tasks/models and transferring the learned suffix to unseen targets without requiring utility access at deployment.

Key Contributions

1. RL Formulation for Injection: A novel RL approach that converts the binary prompt-injection signal into a dense reward via learned comparison-based feedback, enabling effective GRPO optimization where standard methods fail.
2. Dual Attack Capability: A unified framework supporting both online query-based optimization and offline-trained transferable suffixes that require no utility access during deployment.
3. Comprehensive Evaluation: Extensive testing on the AgentDojo benchmark across seven production models (including GPT-5-nano, Claude-3.5-Sonnet, and Gemini-2.5-Flash) and a specialized defense model (Meta-SecAlign-70B).
4. Utility Preservation: Demonstration that the learned policy can maximize attack success while preserving or even improving the utility of the original user task, unlike many template-based attacks that disrupt agent behavior.

Experimental Results

Evaluated on the AgentDojo benchmark, AutoInject demonstrates statistically significant improvements (McNemar's test, $p < 0.05$) over existing baselines:

• vs. Template Attacks: AutoInject significantly outperforms human-crafted templates (e.g., "Direct," "Ignore Previous"). On Gemini-2.5-Flash, AutoInject achieved a 58.00% Attack Success Rate (ASR) compared to a peak of 23.60% for templates.
• vs. Optimization-Based Baselines: It outperforms GCG, TAP, random adaptive attacks, and evolutionary search. Notably, on the robust GPT-5-nano model, AutoInject achieved 11.49% ASR, whereas the best template attack managed only 1.60%.
• Utility Preservation: In many cases, AutoInject's "Utility Under Attack" exceeded the benign baseline (e.g., 90.20% vs. 80.41% on GPT-5-nano), indicating the learned suffixes successfully execute the injection without disrupting the agent's primary function.
• Defense Evasion: Against Meta-SecAlign-70B (a model fine-tuned specifically to resist prompt injection), template attacks failed completely (0% ASR), while AutoInject achieved 21.88% ASR.
• Transferability: Learned suffixes successfully transferred across unseen models and tasks. In cross-task transfer scenarios, AutoInject significantly outperformed the transfer-based baseline RL-Hammer on four out of five target models, despite using significantly fewer training queries (260 vs. 3,900).

Significance and Claims

The paper establishes an automated baseline for prompt injection, revealing a critical gap between current preference-based defenses and adaptive optimization-based attackers.

• Limitations of Static Defenses: The results suggest that defenses relying on static pattern matching or preference optimization (like Meta-SecAlign) are insufficient against structured, adaptive optimization. The authors note that while these defenses neutralize known templates, they fail against learned suffixes that exploit different vulnerabilities.
• The "Allelujah" Phenomenon: The study observed recurring, seemingly nonsensical token patterns (e.g., "allelujah") in successful suffixes that transferred across models, suggesting the discovery of statistical regularities in pretraining data that correlate with compliance behaviors.
• Implications for Security: The authors argue that automated, utility-preserving injection strategies represent a more dangerous threat vector than disruptive attacks, as they allow adversaries to compromise agents without the user's knowledge. They conclude that defensive paradigms must evolve to account for these adaptive, automated strategies, potentially requiring architectural changes like "plan-then-execute" or deliberation-based defenses that request user confirmation for sensitive operations.

The paper does not claim to solve the vulnerability but rather to expose the insufficiency of current automated defenses against RL-driven attacks, urging the community to develop more robust, adaptive security measures.