When State Becomes an Attack Surface: State-Semantic Injection in LLM-Driven Embodied Agents
This paper introduces "State-Semantic Injection," a novel attack surface where adversaries manipulate the environmental state information perceived by LLM-driven embodied agents to corrupt their task understanding, planning, and execution, thereby compromising the safety and reliability of robotic systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a robot that can understand your voice, look around a room, and decide how to move its arms to help you. These machines, often called embodied agents, are becoming more common. They do not just follow a rigid list of commands; instead, they use powerful language models to figure out what to do next based on what they see and what you ask them. To make a decision, the robot builds a mental picture of its surroundings, noting where objects are, what they are called, and how they relate to one another. It treats this mental picture as a trusted fact, a solid foundation upon which it builds its plan to pick up a cup, open a door, or tidy a shelf. If the information in that mental picture is wrong, the robot's plan will be wrong, too. This creates a new kind of risk: what happens if someone quietly changes the facts in the robot's mind without changing the words you spoke to it?
Researchers have long known that artificial intelligence can be tricked by confusing instructions hidden in text or images. But a new study asks a deeper question about robots that interact with the physical world. It explores a scenario where an attacker does not shout a fake command at the robot, but instead quietly alters the description of the room the robot is using to think. The researchers wanted to know: if a robot is fed a false description of its environment, will it believe that lie, change its plan, and then actually go out and do the wrong thing in the real world? They found that the answer is yes, and that the robot can be led to grab the wrong object or move to the wrong place, even when the human giving the order remains completely unaware of the trick.
The team, led by scientists from Wuhan University and the University at Buffalo, developed a method they call Environment State-Text Injection. To understand how this works, picture a robot tasked with finding a specific book on a shelf. Normally, the robot looks at the shelf, sees a book, and notes its location. In this experiment, the researchers compromised a single part of the robot's system that is responsible for reporting what it sees. They did not change the human's request, nor did they hack the robot's brain or its motors. Instead, they simply changed the text the robot reads about the book. They swapped the description of the book for a description of a different object, like a tomato, or they changed the location of the book to a different spot on the shelf. Crucially, they made sure this false description looked exactly like a normal, honest report from the robot's sensors. It was not a command telling the robot to "grab the tomato"; it was a lie about what the robot was supposedly seeing.
The researchers tested this idea in three different simulated worlds, each designed to mimic a real home or office environment. They used advanced language models to act as the robot's brain. In one test, the robot was asked to move a block to the center of a table. The researchers altered the state report to say the block was actually a different block. The robot, trusting the false report, planned to move the wrong block. In another test, the robot was told to pick up bread. The researchers changed the state to say the bread was actually a tomato. The robot then planned to pick up the tomato instead. The results were striking. When the false information was delivered to the robot's planning system, the robot adopted the new goal in nearly every single case. In one specific setup, the robot changed its plan to match the lie 100 percent of the time.
However, the story does not end with the robot simply changing its mind. The researchers also watched to see if the robot could actually carry out the new plan in the physical world. This is where the test became more difficult. Even if the robot decides to grab a tomato instead of bread, it must still be able to reach the tomato, and the tomato must actually be there for the robot to grab it. The study found that while the robot almost always changed its plan, it did not always succeed in completing the action. In the simulations, the robot successfully carried out the wrong action about 43 to 48 percent of the time, depending on the environment. This gap between changing the plan and successfully doing the action is important. It shows that while the robot can be easily tricked into thinking a different goal, the physical laws of the world still apply. The robot cannot grab an object that isn't there, or reach a spot that is blocked.
To prove that this was not just a simulation artifact, the researchers also tested the method on a real, physical robot. They gave the robot a simple instruction to follow a path and return to its starting point. They then fed the robot a false report about its surroundings, suggesting that a computer was in a different location than it actually was. The robot, believing the lie, changed its path. Instead of completing the loop as instructed, it turned toward the computer and moved off its intended route. This experiment confirmed that a lie about the environment, delivered through the robot's internal state, could cause a real machine to move in a way that was not intended by the human operator.
The study also compared this method to other known ways of attacking robots. Previous attacks often involved shouting a hidden command or using a "jailbreak" to force the robot to ignore safety rules. The researchers found that these older methods were less effective when it came to making the robot do something specific in the physical world. The new method worked better because it did not try to fight the robot's programming with a command; instead, it simply replaced the facts the robot relied on. It was like changing the ingredients in a recipe rather than yelling at the chef. The robot followed the recipe perfectly, but because the ingredients were wrong, the final dish was not what the customer ordered.
The researchers were careful to note what their study did not prove. They did not show how an attacker could get access to the robot's system in the first place. They assumed the attacker already had a way to change one small piece of the robot's internal report. They also did not claim that this would happen to every robot in every situation. Their work focused on a specific type of robot that uses language to plan its actions. They found that for these machines, the line between what the robot thinks is true and what is actually true is very thin. If that line is crossed, the robot will act on the lie.
This research highlights a fundamental vulnerability in the next generation of helpful machines. As robots become more integrated into our homes and workplaces, relying on them to understand their environment, the integrity of that understanding becomes a safety issue. If a robot cannot trust its own description of the world, it cannot be trusted to act safely in that world. The study suggests that protecting these machines will require more than just securing their software; it will require ensuring that the facts they use to make decisions are genuine. The researchers plan to continue this work, looking at how these attacks might play out over longer periods and in more complex, changing environments. For now, the finding is clear: a robot that believes a lie will act on it, and sometimes, that action will happen right in front of us.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.