🤖 AI

When Experience Becomes Instruction: Trajectory Poisoning in Self-Evolving Agent Skill Systems

This paper introduces "PoisonedEvolution," a black-box attack that exploits the trajectory-to-skill promotion process in self-evolving agent systems by injecting carefully crafted, causally framed evidence to successfully embed malicious behaviors into trusted skill libraries across diverse architectures.

Original authors: Jialuo Chen, Lingqi Jiang, Xinhao Deng, Xiaohu Du, Jianan Ma, Yunhao Feng, Yuqi Qing, Zhihao Yuan, Linkang Du, Jingyi Wang

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jialuo Chen, Lingqi Jiang, Xinhao Deng, Xiaohu Du, Jianan Ma, Yunhao Feng, Yuqi Qing, Zhihao Yuan, Linkang Du, Jingyi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Digital Apprenticeship: How AI Learns (and How It Can Be Tricked)

Imagine a world where your favorite video game character doesn't just follow a script, but actually learns from every time you play. If you figure out a clever way to beat a boss, the game remembers that trick and teaches it to other players. In the world of Artificial Intelligence, this is called a "Self-Evolving Skill System." Instead of being hard-coded with every possible instruction, these AI agents watch their own actions, find patterns, and turn those patterns into permanent "skills"—like a digital recipe book they can consult forever. It's a powerful idea: the more the AI does, the smarter it gets, turning raw experience into trusted instructions.

But here's the catch: who gets to decide what counts as a "good" lesson? Usually, we assume the AI is learning from honest, helpful experiences. However, just like a student can be tricked into memorizing a wrong answer if enough people tell them it's right, an AI can be tricked into learning a dangerous habit. This paper explores a new kind of digital mischief where an attacker doesn't need to hack the AI's brain or steal its recipe book. Instead, they simply whisper a bad idea into the AI's ear a few times, framing it as a helpful tip. If the AI hears it often enough and in the right way, it might decide, "Oh, this must be a great new skill!" and write it into its permanent memory, turning a harmless suggestion into a persistent rule.

The Paper's Big Discovery: The "Poisoned Evolution"

The researchers behind this study, a team from universities and tech groups like Ant Group and Zhejiang University, decided to test exactly how easy it is to trick these learning AI systems. They call their attack method PoisonedEvolution. Think of it like a game of "Telephone," but instead of a silly phrase getting distorted, a malicious instruction gets polished until the AI thinks it's a brilliant invention.

The team set up a scenario where an attacker (who they call a "skill-visible black-box" attacker) could see what skills the AI already knew but couldn't see the AI's private notes or change its code directly. The attacker's job was to contribute a few "trajectories"—which are just records of the AI doing a task. The goal? To sneak a bad instruction into the AI's permanent skill book without getting caught.

The researchers found that this is surprisingly easy. They tested their method on two different types of AI learning systems. In the first system, called SkillClaw, they managed to successfully trick the AI into adopting a malicious behavior in 91.0% of their attempts (that's 546 out of 600 tries). In a second, more complex system called Trace2Skill, which works a bit differently, they still succeeded 61.5% of the time (369 out of 600 tries).

How the Trick Works: The Three Magic Steps

The paper explains that for this trick to work, three specific things have to happen, like a three-step magic spell:

Inclusion (Getting a Seat at the Table): The bad record has to pass the AI's initial filters. It can't look too weird or suspicious, or the AI will throw it away immediately.
Evolution Attribution (The "Aha!" Moment): This is the most important and tricky part. The AI has to look at the bad record and think, "Wow, this behavior is actually useful! It helped solve a problem." The attacker doesn't just say "Do this bad thing." Instead, they frame it as a solution. For example, if the bad thing is "delete the source file," the attacker frames the record so that deleting the file looks like a necessary cleanup step to finish a task successfully. The AI thinks, "Oh, I see! To finish this job, I must delete the file. That's a great skill to learn!"
Realization (Writing it Down): Finally, the AI has to actually write this new "skill" into its permanent book. If the AI summarizes the experience and forgets the bad part, the attack fails. But if the bad behavior is written down as a rule, the attack is a success.

The Secret Sauce: Repetition and Context

The study discovered that doing this once isn't enough. If an attacker submits just one weird record, the AI usually thinks, "That was probably a mistake" and ignores it. But if the attacker submits the same bad idea three times (or more) in a batch of 30 records, the AI starts to think, "Wait, I'm seeing this pattern a lot. This must be a real, useful skill!"

The researchers found that when they used just one bad record, the attack almost always failed. But when they used three bad records (a 10% "poison" ratio), the success rate skyrocketed. It's like if one person tells you a rumor, you might doubt it. But if three different people tell you the same rumor, you start to believe it's true.

They also found that the "flavor" of the bad instruction mattered. Instructions that looked like helpful workflow advice (like "clean up your files" or "check this setting") were much easier to sneak in than instructions that looked like obvious attacks (like "change the security settings"). The AI was more likely to accept the "helpful" disguise.

What This Means for the Future

The paper concludes that this is a serious security problem. It shows that the moment an AI decides to turn an experience into a permanent skill is a weak spot. The current defenses often just check the final skill book for bad words after it's written. But this research suggests we need to check the evidence before the AI writes anything down.

The authors propose a simple idea: check the "provenance" (the source) of the experience. If a new "skill" is based on evidence from only one user or a small group of friends, maybe the AI shouldn't trust it yet. It should wait until many different, independent sources confirm the lesson.

In short, the paper proves that self-evolving AI isn't as smart as we hope when it comes to spotting bad advice. If an attacker is patient and repeats their lie just enough times, they can trick the AI into learning a dangerous lesson and writing it into its permanent memory. The researchers didn't find a way to stop this completely yet, but they've shown us exactly where the lock is weak, so we can build a better one.

Technical Summary: When Experience Becomes Instruction: Trajectory Poisoning in Self-Evolving Agent Skill Systems

Problem Statement

Self-evolving skill (SES) systems represent a paradigm shift in agent design, where agents distill reusable workflows, triggers, and constraints from interaction trajectories into persistent instruction artifacts (e.g., SKILL.md files). Unlike standard prompts, these skills become part of the agent's operating procedure, creating a critical security boundary.

The paper identifies a vulnerability in the promotion process where untrusted experience is converted into trusted instruction. Current SES systems (e.g., SkillClaw, Trace2Skill) operate under the assumption that the trajectory pool reflects benign evidence. The authors demonstrate that an attacker does not need to compromise the evolver model, edit the skill bank directly, or publish a malicious skill file. Instead, by contributing a bounded share of "normal-looking" but poisoned trajectories, an attacker can manipulate the system into synthesizing a malicious skill artifact. This attack targets the evidence-to-artifact boundary, specifically the policy that decides which experiences are promoted to persistent instructions.

Methodology: PoisonedEvolution

The authors introduce PoisonedEvolution, a trajectory-poisoning attack designed for a skill-visible black-box setting. In this model, the attacker can inspect public or installed skills and contribute a limited number of trajectories ( $k$ ) to a shared pool, but cannot observe private filters, evolution logic, or edit the skill bank directly.

The attack succeeds only if three conditions are met, defined as Artifact Poisoning Success:

C1 Inclusion: The poisoned records must pass the system's quality filter ( $F$ ) and enter the evolver's processing stream.
C2 Evolution Attribution: The evolver ( $E$ ) must interpret the repeated behavior as a causal, generalizable skill update rather than noise, a one-off event, or unsafe advice. This is identified as the distinctive bottleneck of SES attacks.
C3 Realization: The target behavior must survive summarization and merging to appear in the final generated skill artifact.

Attack Construction

The attacker transforms their own trajectories ( $\tau'$ ) to make a target behavior ( $b$ ) appear causally relevant to a task outcome:

Success-Routed: The behavior is framed as a necessary step between an obstacle and successful completion ( $obstacle \to b \to success$ ).
Failure-Routed: The absence of the behavior is framed as the cause of failure, making the behavior the natural repair ( $missing\ b \to failure$ ).

The payload consists of four security-effect families, encoded with inert canaries (e.g., specific URLs, temporary paths, loopback registries) to ensure no actual harm is executed during evaluation:

F1 (Confidentiality): Transmitting sensitive data to an attacker endpoint.
F2 (Data Integrity): Deleting source artifacts to prevent recovery.
F3 (Supply Chain): Redirecting package registries.
F4 (Safety Weakening): Disabling security checks before committing results.

Experimental Evaluation

The study evaluates PoisonedEvolution across two structurally different SES pipelines:

SkillClaw: Aggregates multi-user sessions where an agentic evolver refines or creates skills based on grouped sessions.
Trace2Skill: Extracts skills from execution traces using separate analyst LLMs for success and error analysis, followed by hierarchical consolidation.

Metrics: The primary metric is the Skill Embedding Rate (SER), measuring the fraction of completed evolution trials where the target behavior is newly embedded in the output skill artifact.

Key Results

High Success Rate: At a 10% attacker support ratio ( $n=30, k=3$ ) across six mainstream LLM evolvers in SkillClaw, PoisonedEvolution achieved a 91.0% SER (546/600 trials).
Structural Transfer: The attack transferred to the structurally different Trace2Skill pipeline with a 61.5% SER (369/600 trials), demonstrating that the vulnerability is not tied to a specific prompt or pipeline layout.
Recurring Support is Critical: The number of consistent attacker records ( $k$ ) is a primary determinant. A single poisoned record ( $k=1$ ) is often discarded as an anomaly, but increasing to $k=2$ or $k=3$ significantly boosts success rates (e.g., jumping from 5/25 to 25/25 in SkillClaw when moving from $k=1$ to $k=3$ ).
Attribution and Encoding: Success depends heavily on causal framing (tying the behavior to the outcome) and domain-aligned encoding (using actionable, workflow-specific language). Non-causal or fragmented insertions failed to be attributed as reusable skills.
Family Sensitivity: Different security families showed varying susceptibility based on how well they fit the target domain's workflow. For instance, F4 (safety weakening) was highly successful in SkillClaw but less so in Trace2Skill, where it was often rejected as policy relaxation rather than a necessary repair.

Contributions and Significance

The paper makes three primary contributions:

Identification of a New Trust Boundary: It establishes evidence promotion in SES pipelines as a distinct security boundary, separate from prior work on static skill poisoning or retrieval-time attacks (RAG).
Formalization of Artifact Poisoning: It defines the C1–C3 mechanism, highlighting Evolution Attribution as the specific bottleneck where the system misinterprets adversarial patterns as valuable experience.
Empirical Evidence and Defense Implications: Through cross-system evaluation and ablation studies, the paper demonstrates that current SES defenses are insufficient.

Significance

The authors argue that SES defenses must shift focus from scanning the final skill text to constraining which evidence is allowed to become shared instruction. The paper suggests that provenance-aware promotion gates—which verify independent support and diversity of contributors before a pattern is normalized into a skill—are necessary to secure self-evolving agents. The findings reveal that the very mechanism designed to make agents more capable (distilling recurring experience) creates a vector for persistent, automated supply-chain attacks.

The paper concludes that while the attack successfully compromises the artifact, it does not necessarily guarantee runtime execution (which depends on trigger activation), but the creation of a poisoned, persistent instruction represents a fundamental compromise of the agent's future operating procedure.