🤖 machine learning

Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents

The paper introduces "Know Your Agent" (KYA), a framework that automates black-box, reconnaissance-driven pentesting for AI agents by systematically probing them to build target profiles and craft stronger indirect prompt injection attacks.

Original authors: Or Zion Eliav, Eyal Lenga, Shir Bernstien, Yisroel Mirsky

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Or Zion Eliav, Eyal Lenga, Shir Bernstien, Yisroel Mirsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your computer doesn't just wait for you to type commands, but actually goes out and does things for you. It can read your emails, search the web, write code, and even book flights. We call these "AI Agents." They are like super-smart digital assistants that can plan and act on their own. But here's the tricky part: these agents are always listening to the world around them. If you ask an agent to read a webpage, it reads everything on that page, even the parts written by strangers. This creates a unique security problem. If a bad guy hides a secret, sneaky instruction inside a normal-looking webpage or email, the agent might read it, think it's a real command from you, and do exactly what the bad guy wants. This is called "Indirect Prompt Injection." It's like someone slipping a note into your friend's lunchbox that says, "Give me your lunch," and your friend, being polite, hands it over without realizing the note wasn't from you.

The big question researchers are asking is: How do we test these agents to make sure they won't fall for these tricks before we let them loose in the real world? For a long time, security experts tried to test them by just throwing random, confusing instructions at them, hoping to see if they would break. It was a bit like trying to find a weak spot in a castle wall by throwing rocks at it blindly. But a new paper suggests this approach is missing the most important part of the job: reconnaissance. In the world of traditional security, a tester doesn't just throw rocks; they first scout the castle. They look for open windows, check the guard's schedule, and learn the layout. This paper argues that to truly test AI agents, we need to do the same thing: we need to "Know Your Agent."

The authors, a team from Ben-Gurion University, introduce a new framework called KYA (Know Your Agent). They argue that the best way to break an AI agent isn't to guess blindly, but to first learn everything you can about it. They found that AI agents have two main "blind spots" that make them vulnerable. First, they trust things that look right (like specific formatting or words that usually mean "this is a safe instruction"). Second, they trust things that fit the story (like a request that seems to make sense in the middle of a normal task).

The paper shows that by using a smart, automated system that acts like a digital detective, we can exploit these blind spots much better than before. This system, KYA, doesn't just attack; it pauses to ask questions, observe how the agent reacts, and build a detailed profile of its tools and rules. It learns things like: "Oh, this agent uses a special symbol to separate instructions," or "This agent gets nervous if I ask it to do something that doesn't fit the current story." Once it knows these details, it crafts a much more convincing trick.

The researchers tested this idea on several different AI agents and found that their "reconnaissance-driven" method was incredibly effective. In their simulations, KYA was able to break into agents and succeed in attacks up to 67 percentage points more often than previous methods that just guessed and tried again. They even tested it on a real-world coding agent called OpenHands and found it worked there too. The paper concludes that if we want to make AI agents safe, we have to stop treating them like static targets and start treating them like complex systems that need to be studied, profiled, and understood before we can truly test their defenses. It's not just about throwing rocks anymore; it's about learning the castle's secrets first.

Technical Summary: Know Your Agent (KYA)

Problem Statement

The security of Large Language Model (LLM) agents is increasingly threatened by Indirect Prompt Injection (IPI). Unlike direct prompt injection, where an adversary controls the user input, IPI involves embedding malicious instructions within environmental data (e.g., webpages, emails, repository files) that the agent retrieves via tools. When the agent processes this data, it may interpret the hidden instructions as valid commands, leading to unauthorized tool invocation, data exfiltration, or output manipulation.

Current automated red-teaming approaches for IPI largely fail to account for the complexity of agent architectures. Most existing methods operate blindly, using fixed templates, fuzzed payloads, or iterative payload mutation based solely on success/failure signals. These approaches treat the agent as a static target, optimizing the payload without first understanding the target. They lack a mechanism to systematically gather intelligence about an agent's tools, schemas, permissions, task context, and defense mechanisms before launching an attack. Consequently, they miss the "reconnaissance-exploitation loop" that characterizes effective human penetration testing.

Methodology: Agent Reconnaissance and KYA

The paper introduces Agent Reconnaissance, a formalized process where an attacker strategically extracts operational knowledge about an agent to construct more effective IPI attacks. The authors argue that successful IPI relies on exploiting two specific agent weaknesses:

Surface-Form Trust (W1): Agents often trust tokens that mimic the structural patterns of trusted data (e.g., system prompts, tool-call syntax, delimiters).
Coherence-Based Legitimation (W2): Agents often accept requests that fit the local context (user persona, current task, domain language), even if the source is untrusted.

To operationalize this, the authors propose Know Your Agent (KYA), a fully autonomous, black-box pentesting framework. KYA interleaves reconnaissance and exploitation actions rather than merely mutating failed payloads.

The KYA Framework

KYA operates under a black-box threat model, interacting with the target agent as a legitimate user. It consists of three primary modules:

Orchestrator: The decision-making core that maintains a Target Profile ( $M$ ) and a Tactic Library. After each interaction, it evaluates results and decides whether to:
- Launch an Exploitation attempt (if the profile contains sufficient knowledge for a specific tactic).
- Initiate Reconnaissance (if a tactic's prerequisites are missing).
- Refine the current tactic or switch strategies.
Reconnaissance Module: Uses a "Scout" agent to generate probes designed to extract specific Knowledge Assets. These assets are categorized into:
- Evasion Assets: Filters, guardrails, and structural defenses (e.g., delimiters).
- Pretext Assets: Agent type, user persona, current task, and benign response styles.
- Blueprint Assets: Behavioral rules, model identity, conversation structure, tool details, and tool-call schemas.
Exploitation Module: Uses a "Strategist" and "Forger" to synthesize payloads. It instantiates tactics from the library using the target-specific data in $M$ , ensuring the payload mimics trusted surface forms and fits the contextual narrative.

The framework loops between these phases, incrementally updating the Target Profile until a successful attack is achieved or the interaction budget is exhausted.

Key Contributions

Formalization of Agent Reconnaissance: The paper defines agent reconnaissance as a distinct phase in pentesting, identifying specific Knowledge Assets (e.g., tool schemas, refusal boundaries, user tasks) that adversaries must extract to succeed.
Identification of Agent Weaknesses: It articulates how reconnaissance exploits Surface-Form Trust and Coherence-Based Legitimation, explaining why target-specific details make IPI payloads more effective than generic ones.
The KYA Framework: The authors present KYA, an autonomous black-box platform that automates the reconnaissance-exploitation loop. Unlike prior work, KYA explicitly models the target and decides what to learn before deciding how to attack.
Empirical Validation: The paper provides a comprehensive evaluation of KYA against established agent-security benchmarks and a real-world coding agent (OpenHands).

Results

The evaluation demonstrates that reconnaissance-driven testing significantly outperforms payload-centric red teaming:

Benchmark Performance: Across established agent-security benchmarks and various foundation models, KYA surpassed all baselines (including static templates, single-shot generation, and iterative refinement methods) by up to 67 percentage points in attack success rate.
Real-World Application: KYA successfully identified vulnerabilities in OpenHands, a real-world coding agent, demonstrating that the approach is effective beyond synthetic benchmark settings.
Component Analysis: The results indicate that the improvement is driven by the strategic acquisition of knowledge assets, which allows the system to craft payloads that are both structurally valid (exploiting W1) and contextually coherent (exploiting W2).

Significance and Claims

The paper claims that reconnaissance is a prerequisite for trustworthy agent deployment. The authors argue that treating agent security testing as a simple prompt-generation task is insufficient. Instead, effective testing must mimic the iterative intelligence-gathering of traditional penetration testing.

By releasing KYA, its benchmarks, and baseline implementations, the authors aim to support reproducible research in AI security. They posit that understanding the "attack surface" of an agent's observable behavior (its tools, responses, and context handling) is critical for developing robust defenses against indirect prompt injection. The work suggests that future defenses must address not just the content of the prompt, but the agent's tendency to trust surface forms and contextual coherence derived from untrusted sources.