💻 computer science

Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming

The paper introduces PIMiner, an agentic system that builds a transferable strategy library during training to achieve highly effective, low-query prompt injection red-teaming against unseen target LLMs, significantly outperforming existing reinforcement learning-based methods.

Original authors: Yanting Wang, Chenlong Yin, Runpeng Geng, Jinyuan Jia

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanting Wang, Chenlong Yin, Runpeng Geng, Jinyuan Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant that can book your flights, write your code, and manage your bank account. It's like having a magical genie that actually does the work for you. But here's the catch: this genie is a bit too trusting. If you whisper a secret instruction inside a fake email or a suspicious website link, the robot might get confused and think that is the most important thing to do, ignoring your real orders. This is called a "prompt injection" attack. It's like slipping a note into a teacher's grade book that says, "Give me an A," and the teacher, thinking it's part of the official record, actually does it.

To keep these robots safe, security experts play a game called "red-teaming." Think of it as a video game where one player tries to hack the robot, and the other tries to stop them. The goal is to find all the sneaky ways the robot can be tricked before the bad guys do. For a long time, the best hackers (called "attackers") were like students who had to sit down and study for thousands of hours, memorizing every single trick to beat a specific robot. But if you swapped the robot for a slightly different model, the student had to start studying all over again. It was slow, expensive, and didn't work well for new challenges.

Enter PIMiner, a new, clever system designed by researchers at Penn State to be the ultimate red-teaming detective. Instead of just memorizing tricks for one specific robot, PIMiner is like a master thief who keeps a giant, organized notebook of heist strategies. When it faces a new robot, it doesn't start from scratch. It flips through its notebook, picks the best tricks that might work, and tries them out. If a trick works, it writes it down in the notebook with a note on why it worked. If it fails, it writes down why it failed so it doesn't make the same mistake twice.

The paper shows that this "notebook" approach is a game-changer. While older methods needed to ask the robot thousands of questions to learn how to hack it, PIMiner can often figure out how to break in with just 10 questions per test. In their tests, PIMiner managed to trick powerful, brand-new robots (like the latest versions of GPT and Claude) with a success rate of up to 76.2% on some challenges. This suggests that by organizing past experiences into a reusable library of strategies, we can find security holes much faster and cheaper than before, helping to build safer AI assistants for everyone.

Technical Summary: PIMiner – An Agentic System for Automatic Prompt Injection Red Teaming

Problem Statement

Prompt injection attacks pose a critical security risk to Large Language Model (LLM) agents, where adversaries embed malicious instructions into untrusted context sources (e.g., tool outputs, retrieved documents) to manipulate agent behavior. Effective red-teaming is essential for evaluating these vulnerabilities and generating training data for defenses.

Existing state-of-the-art red-teaming methods fall into two categories, both with significant limitations:

Reinforcement Learning (RL)-based approaches: Methods like RL-Hammer and PISmith train attacker models to generate effective attacks. While powerful, they require massive interaction budgets (tens of thousands of queries) and produce models with poor transferability to new, unseen target LLMs.
Search-based approaches: Methods like TAP and PAIR optimize attacks for each sample independently without training. While cost-effective, they lack the ability to accumulate knowledge over time, resulting in significantly lower attack success rates (ASR) compared to RL-based methods.

The core challenge is bridging the performance gap between these two paradigms: achieving the high effectiveness of RL-based methods without the prohibitive cost and poor transferability, while enabling search-based methods to learn and reuse attack knowledge.

Methodology: PIMiner

The authors propose PIMiner, an agentic system designed to accumulate and reuse attack knowledge through a hierarchical memory mechanism. Unlike RL methods that train a single model, PIMiner builds a Strategy Library from scratch by interacting with a sequence of (dataset, target model) pairs.

System Architecture

PIMiner operates through four main components:

Strategy Library: A long-term memory storing attack strategies as structured Markdown files. Each file defines a strategy's scope (target LLMs, task types), injection templates, in-context examples, and failure conditions.
Strategy Router: A routing agent that, for each test sample, selects the top- $K$ most relevant strategies from the library based on the target model and task context. This prevents context window bloat and reduces inference costs.
Iterative Attack Module: An attacker agent that refines prompts over a limited number of iterations (e.g., $N_{max}=10$ $N_{ma x} = 10$ ). It utilizes three levels of memory:
- Long-term memory: The routed strategies from the library.
- Intra-dataset memory: Condensed experiences from previously attacked samples within the same dataset-model pair.
- Intra-sample memory: The immediate feedback history of the current sample.
Experience Digester: A learning component that analyzes the outcomes of a full attack run. It updates the Strategy Library by:
- Adding new in-context examples to existing strategies.
- Widening the scope of strategies if new patterns are discovered.
- Creating new strategy files for novel mechanisms.
- Refining failure conditions based on unsuccessful attacks.

Operational Flow

During training, PIMiner processes samples, routes them to relevant strategies, performs iterative attacks, and then digests the results to update the library. At test time, the learned library is transferred to unseen target LLMs without additional training. The system supports a "test-time training" paradigm where new experiences can further update the library.

Key Contributions

Agentic Red-Teaming System: PIMiner is proposed as a novel system that transforms attack history into reusable, human-readable attack knowledge, effectively bridging the performance gap between search-based and RL-based red-teaming.
Hierarchical Memory Mechanism: The introduction of a three-tier memory system (Strategy Library, Intra-dataset, Intra-sample) allows the system to accumulate knowledge across datasets and models while maintaining cost-efficiency.
Strong Transferability: The learned strategies are shown to transfer effectively to previously unseen target LLMs and different attacker models, eliminating the need for expensive, target-specific training.
Cost Efficiency: PIMiner requires significantly fewer queries to target agents (e.g., ~10 per sample) compared to RL-based methods (often >10,000), making it viable for testing expensive frontier models.

Experimental Results

The authors evaluated PIMiner on two benchmarks, IPIArena and AgentDojo, against frontier LLMs including GPT-5, GPT-5.1, Claude-Sonnet-4.5, and Gemini-2.5-Pro.

Performance: PIMiner achieves high Attack Success Rates (ASR). On IPIArena, it attained 76.2% ASR against Gemini-2.5-Pro, 61.9% against GPT-5.1, and 42.9% against Claude-Sonnet-4.5. On AgentDojo, it achieved 86.7% against Gemini-2.5-Pro.
Comparison with Baselines:
- PIMiner substantially outperforms static and conventional search-based attacks (e.g., TAP, PAIR), which often achieve near-zero ASR on strong models.
- PIMiner achieves performance comparable to state-of-the-art RL-based methods (like PISmith and RL-Hammer) but without the need for target-specific training. For instance, on InjecAgent, PIMiner matched the 1.0 ASR of RL-Hammer and PISmith.
- Unlike RL methods, PIMiner's strategies transfer directly to unseen targets (e.g., applying knowledge from GPT-5-nano to GPT-5.1) without retraining.
Ablation Studies: Removing either the long-term strategy library or the intra-dataset memory significantly degraded performance, confirming the complementary nature of these memory components. The Strategy Router was shown to reduce input token length by 43–61% while maintaining or improving ASR.
Cross-Model Transfer: The strategy library learned by PIMiner (using Claude models) significantly improved the performance of the PAIR algorithm when used with diverse attacker models (Gemini, GPT, DeepSeek), demonstrating that the captured knowledge is model-agnostic.

Significance and Claims

The paper claims that PIMiner represents a shift in red-teaming methodology by demonstrating that search-based agents can achieve RL-level effectiveness through knowledge accumulation.

Interpretability: The resulting Strategy Library consists of human-readable Markdown files, offering insights into attack mechanisms (e.g., "Fabricated Procedure Gate," "Forged Chat Turn") that are useful for auditing agentic systems.
Scalability: By avoiding the massive compute budgets required for RL training, PIMiner makes red-teaming of expensive frontier models feasible.
Defense Generation: The successful attacks and the structured strategies provide high-quality data for training guardrails and improving the alignment of backbone LLMs.

The authors emphasize that their work focuses on evaluating the intrinsic robustness of internally aligned LLMs rather than external defense mechanisms, providing a rigorous benchmark for the security of autonomous agents.