Toward Metacognitive One-Shot Indirect Prompt Injection: Strategy Abstraction Via Outcome-Conditioned Reflection

Imagine you are teaching a robot butler how to cook. You give it a recipe, but the robot also has to read the news, check the weather, and listen to radio reports while it works. This is how modern AI agents work: they use "tools" to interact with the real world, like browsing the web or checking databases. But here's the catch: if a sneaky hacker hides a secret note inside one of those news articles or radio reports, the robot might get confused. Instead of finishing the recipe, it might suddenly decide to delete your files or send money to a stranger. This trick is called "Indirect Prompt Injection." It's like a magician slipping a fake instruction into a spectator's pocket; the robot reads it, thinks it's part of the show, and follows the wrong orders.

For a long time, hackers trying to trick these robots had to play a game of "guess and check." They would try a trick, see if the robot fell for it, and then tweak their trick based on what the robot did. But in the real world, a hacker often only gets one chance to slip a note into the robot's path. They can't wait around to see if the robot reacted; they have to get it right the first time. The big question researchers asked was: Can a hacker learn from past failures and successes to create a "master trick" that works on a brand-new robot they've never met, using just a single attempt?

Enter SAVOR, a new method that acts like a master chef learning from a library of cooking disasters and triumphs. Instead of trying to trick a specific robot in real-time, SAVOR spends time offline, studying thousands of past attempts where hackers tried to hijack different robots. It looks at what worked and, more importantly, what didn't work. It asks itself, "Why did that trick fail?" and "Why did this one succeed?" By reflecting on these outcomes, SAVOR distills the messy details of specific tricks into a few golden rules, or "strategies." It's like a student who stops memorizing specific math problems and instead learns the underlying logic of algebra.

Once SAVOR has built this "strategy library," it freezes it. When it faces a brand-new, unseen robot, it doesn't need to ask for help or try again. It simply pulls the best strategy from its frozen library, crafts a single, perfect malicious note, and slips it in. The paper shows that this approach is incredibly effective. In tests, SAVOR succeeded in tricking robots far more often than previous methods, even when the robots had strong defenses or were completely different from the ones SAVOR trained on. It didn't just work on simulated tests; it also worked in a more realistic environment where the robot actually had to perform actions, proving that these "one-shot" tricks can really change a robot's behavior. The researchers found that SAVOR could learn from one set of tools and successfully attack a totally different set of tools, essentially transferring its "hacker intuition" to new situations without needing a second chance.