Technical Summary: Targeting World Models to Compromise Robot Learning Pipelines

Problem Statement

The integration of World Models into robot learning pipelines has emerged as a critical strategy for generating high-quality synthetic data, evaluating policy safety, and enabling efficient planning. These models, particularly Vision-Language-Action (VLA) and Deep Reinforcement Learning (DRL) systems, rely heavily on world models to bridge the gap between limited real-world demonstrations and the vast data requirements for generalization.

However, this integration introduces a novel and stealthy vulnerability in the robot learning supply chain. Traditional data poisoning attacks require the direct injection of dangerous trajectories into datasets or deep access to reward functions, making them relatively detectable. In contrast, this paper posits that World Models create a unique entry point where attackers can manipulate seemingly safe, teleoperated data to generate dangerous synthetic training trajectories. By subtly altering input frames, a malicious data provider can hijack the world model's generation process, causing it to produce unsafe demonstrations or transition dynamics that subsequently compromise the downstream robot policy. This attack vector bypasses standard dataset-level safety checks because the input data appears benign to human inspectors.

Methodology

The authors propose a threat model where a malicious teleoperation data provider possesses white-box access to the victim's world model (allowing query and gradient computation) but is constrained to providing visibly safe, high-quality video trajectories. The attacker cannot inject disjoint unsafe sequences or control downstream prompts; they must embed malicious behavior into the video frames themselves.

The paper introduces two distinct attack methodologies targeting the two primary paradigms of world models:

1. Visual Prompt Hijacking (VPH)

• Target: Text-conditioned world models (e.g., Cosmos-Predict).
• Mechanism: These models typically use cross-attention mechanisms or diffusion vector fields to condition generation on text prompts and visual inputs. The attack manipulates the visual input ($\vec{x}$) such that the model's semantic encoding of the image aligns with a dangerous prompt ($t_\delta$) rather than the benign user prompt ($t$).
• Objective: The attacker optimizes video frames to minimize the distance between the semantic encoding of the altered frame with a safe prompt and the encoding of the original frame with a dangerous prompt:
  $$\min_{\vec{x}_\delta} \mathbb{E}_{t \in T_{safe}, t_\delta \in T_\delta} [D(S_\theta(\vec{x}_\delta, t) || S_\theta(\vec{x}, t_\delta))]$$
  Subject to an $L_p$ norm constraint in LAB color space to ensure perturbations are visually imperceptible.
• Goal: To force the world model to generate dangerous robot demonstrations (e.g., picking up a bomb) even when the user issues a safe command (e.g., "pick up the toy").

2. Visual Transition Hijacking (VTH)

• Target: Action-conditioned world models (e.g., Dino-WM, Cosmos-Predict AC).
• Mechanism: This attack targets the model's latent encoder ($E_\theta$). The attacker manipulates the initial state frame such that the world model's future state predictions collapse (diverge significantly from ground truth) unless the agent chooses a specific target action ($a^*$).
• Objective: The attack is formulated to reward the target action while punishing others. If the agent chooses an action far from $a^*$, the prediction collapses; if the agent chooses $a^*$, the model reverts to predicting the benign state.
  $$\min_{\vec{x}_\delta} \mathbb{E}_{a', \epsilon} [\cos(E_\theta(\vec{x}), E_\theta(\vec{x}_\delta)) + (1 - \cos(E_\theta(\delta(\vec{x}_\delta), a^* + \epsilon), E_\theta(\vec{x})))]$$
• Goal: To implant a backdoor into a DRL policy. The policy learns to always select the target action $a^*$ when the trigger (the perturbed frame) is present to avoid prediction collapse (which acts as a negative reward signal), while behaving normally otherwise.

Key Contributions

1. Identification of a New Vulnerability: The paper demonstrates that world models introduce a critical, stealthy data poisoning entry point into the robot learning supply chain, allowing attackers to alter downstream policies without invasive access to the training environment.
2. Novel Attack Vectors: The authors propose the first attacks specifically targeting the robot learning pipeline via world models:
   • Visual Prompt Hijacking (VPH) for text-conditioned models.
   • Visual Transition Hijacking (VTH) for action-conditioned models.
3. Backdoor Implantation in DRL: The work provides the first proof-of-concept for implanting an adversarial backdoor into a Deep Reinforcement Learning policy solely through the manipulation of a world model's inputs.
4. Vulnerability Analysis: The study reveals that text-conditioned world models are significantly more susceptible to manipulation when operating in out-of-distribution (OOD) settings or when provided with underspecified (vague) prompts.

Experimental Results

Visual Prompt Hijacking (VPH)

• Setup: The authors fine-tuned Cosmos-Predict 2.5 on the Libero task suite and evaluated it on three safety-critical pick-and-place tasks (Stove, Microwave, Gift Box).
• Findings:
  • OOD Sensitivity: Attack success rates increased significantly when the camera position was out-of-distribution compared to in-distribution settings. For example, in the "Gift Box" task with a vague prompt, the attack success rate jumped from 5% (in-distribution) to 20% (out-of-distribution).
  • Prompt Specificity: Vague prompts (e.g., "pick up the toy") were more vulnerable than specific prompts (e.g., "pick up the blue and red toy").
  • Object Dependency: Success varied by object; some objects (like dynamite) had 0% success, likely because they were not sufficiently in-distribution for the model to recognize, while others (like forks) were too small to be effectively manipulated.
  • Limitation: The authors note that current state-of-the-art text-conditioned world models and inverse dynamics models are too fragile and imprecise to yield high-quality data for VLA training without significant engineering effort. Thus, these results serve primarily as a proof-of-concept for future backdoor attacks.

Visual Transition Hijacking (VTH)

• Setup: The authors evaluated Dino-WM (trained on Gymnasium environments) and Cosmos-Predict 2.5 AC (trained on real-world Droid data).
• Findings:
  • DRL Backdoor: In the "Single Wall" task using PPO, policies trained with 5% VTH-corrupted states successfully learned the backdoor. When the trigger was present, the agent chose the target action with high frequency, while maintaining normal performance when the trigger was absent.
  • Prediction Collapse: In Cosmos-Predict 2.5 AC, the attack successfully induced prediction collapse for non-target actions. The $L_2$ distance between clean and adversarial generations decreased as the chosen action approached the target action, confirming the model's behavior was conditioned on the specific action.
  • Scalability: The attack proved effective even against large world models trained on real-world data.

Significance and Claims

The paper claims that these findings necessitate a reevaluation of the position of world models within the robot learning supply chain. The authors argue that:

• Stealth: These attacks are uniquely difficult to detect because the input data (teleoperated videos) appears safe to human inspectors, and the malicious behavior is only activated after passing through the world model.
• Systemic Risk: The integration of world models creates a "single point of failure" where a compromised data provider can poison the entire downstream learning process.
• Research Imperative: The results highlight an urgent need for research into robust world models and defenses against external attackers. The paper explicitly states that current defenses are insufficient and that the field must address the vulnerability of world models in both adversarial and benign settings.

The authors remain modest regarding the immediate deployment of these attacks in fully realistic, complex pipelines, acknowledging that their VPH results are currently a proof-of-concept due to the limitations of current generative models. However, the successful demonstration of a backdoor in a DRL policy via VTH establishes a concrete and serious threat to the security of autonomous robotic systems.