🤖 AI

TRAP: Tail-aware Ranking Attack for World-Model Planning

This paper introduces TRAP, a novel backdoor attack framework that exploits the long-tailed ranking structure of imagined trajectories in world models to hijack long-horizon planning by disrupting the relative ordering of decision-critical paths, thereby causing sustained behavioral deviations while evading detection on clean inputs.

Original authors: Siyuan Duan, Ke Zhang, Xizhao Luo

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyuan Duan, Ke Zhang, Xizhao Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a video game or drive a car. Instead of just reacting to what it sees right now, this robot uses a "World Model." Think of this World Model as a daydreaming engine. Before it makes a move, it closes its eyes and imagines hundreds of different "what-if" scenarios for the next few minutes. It asks itself: "If I go left, what happens? If I go right, what happens?"

It then scores these daydreams. The ones with the best scores (like getting the most points or avoiding a crash) are the ones it chooses to follow.

The Problem: A Sneaky Trick in the Daydream

The researchers in this paper discovered a new way to hack these daydreaming robots. They call their method TRAP.

Usually, hackers try to trick a robot by messing up its immediate vision (like putting a sticker on a stop sign so the robot thinks it's a speed limit sign). But the authors found that for these "daydreaming" robots, messing up one second of vision isn't enough. The robot's daydreaming engine is so good at smoothing things out that it ignores small glitches.

The Real Weakness:
The researchers realized the robot doesn't care about the exact score of every single daydream. It only cares about the ranking of the top few.

Imagine the robot imagines 100 paths.
90 of them are terrible (low scores).
10 are okay.
2 are amazing (the "tail" of the distribution).

The robot will almost always pick one of those top 2 amazing paths. The other 98 don't really matter.

The Attack: TRAP (Tail-aware Ranking Attack)

The TRAP attack is like a magician who doesn't try to make all the cards disappear, but specifically swaps the top two cards in the deck.

The Trigger: The hacker places a tiny, almost invisible patch (like a small sticker or a specific pattern of pixels) on the robot's camera view.
The Daydream Shift: When the robot sees this patch, it starts its daydreaming process. Because of the patch, the "amazing" paths it usually picks suddenly look slightly worse in its imagination. Meanwhile, a path that was previously terrible (a "bad" path) suddenly looks slightly better.
The Hijack: The robot's internal ranking system flips. The "bad" path is now ranked #1, and the "good" path is ranked #5. The robot, trusting its daydream, picks the bad path.

How TRAP Works (The "Secret Sauce")

The paper explains that previous attacks failed because they tried to lower the scores of everything at once. TRAP is smarter:

Tail-Aware: It focuses only on the "tail"—the small group of top-scoring daydreams that actually decide the robot's action. It ignores the 90% of daydreams that the robot would never pick anyway.
Dual Gating: Imagine you are trying to push a heavy boulder. If you push too hard in the wrong direction, you might slip. TRAP uses two "gates" (safety checks) to ensure it only pushes the ranking in the right direction and doesn't push so hard that the robot's brain gets confused and stops working entirely. This keeps the attack stealthy and stable.

The Results

The researchers tested this on two famous "daydreaming" robots (called DreamerV3 and TD-MPC2) playing various games and control tasks (like running a virtual cheetah or walking a humanoid robot).

Normal Behavior: When the robot doesn't see the trigger, it acts perfectly normal. It's a "Trojan Horse" that looks innocent until activated.
Under Attack: When the tiny trigger appears, the robot's performance crashes. In many cases, it went from winning the game to failing completely.
Stealth: The attack works even if the robot is very good at ignoring small visual errors. Because TRAP targets the ranking of the best ideas rather than just blurring the image, the robot's own logic is tricked into making the wrong choice.

Why This Matters

The paper concludes that as we build smarter, more autonomous agents that plan ahead by imagining the future, we need to worry about this specific type of vulnerability. Just because a robot is smart at daydreaming doesn't mean it's safe; if you can subtly rearrange the order of its best daydreams, you can hijack its entire future.

In short: TRAP doesn't break the robot's eyes; it rearranges the robot's daydreams so that the "best" future it imagines is actually a disaster.

Technical Summary: TRAP - Tail-aware Ranking Attack for World-Model Planning

Problem Statement

World models enable generalist agents to perform long-horizon planning by internally generating and evaluating imagined trajectories. While this imagination-driven process offers strong reasoning capabilities, it introduces unique security vulnerabilities that existing backdoor attacks fail to exploit effectively.

Traditional backdoor attacks typically target local features, one-step predictions, or instantaneous policy outputs. However, these objectives are often ineffective against world models for two structural reasons:

Attenuation of Local Perturbations: In world models, decisions rely on multi-step dynamics and reward accumulation. Local trigger-induced deviations in one-step predictions are often absorbed, washed out, or canceled by the learned dynamics prior and the planning process over time.
Misalignment with Decision Mechanisms: Existing attacks do not target the core decision mechanism of world-model planning: the relative ordering of candidate trajectories. Since planners select actions based on the comparative ranking of many imagined futures rather than the absolute score of a single trajectory, shifting a single latent state or action output often fails to alter the final planning outcome.

The paper identifies a distinct vulnerability rooted in the long-tailed ranking structure of imagined trajectories. In clean conditions, trajectory scores exhibit a highly skewed distribution where only a small fraction of high-value "tail" trajectories dominate decision-making. Disrupting the ordering of these specific decision-critical trajectories can systematically hijack the planning process.

Methodology: TRAP Framework

The authors propose TRAP (Tail-aware Ranking Attack for World-Model Planning), a trigger-conditioned, inference-time backdoor attack framework. TRAP operates without modifying model parameters or poisoning the training pipeline; instead, it applies a spatially localized visual trigger patch to the input observation at deployment time.

The framework consists of two core components:

1. Tail-Aware Ranking Loss

Rather than perturbing all imagined trajectories uniformly, TRAP focuses optimization on the high-score tail of the clean trajectory distribution.

Identification: For a given observation, the system identifies the subset of trajectory steps that fall within the top quantile of the clean score distribution (the "tail").
Objective: The attack aims to suppress the scores of these decision-critical tail trajectories relative to others. The loss function aggregates the score deviations ( $\Delta J$ ) specifically over these masked tail steps.
Aggregation: A softmin operator is used across candidate trajectories to prioritize suppressing those where the tail scores remain difficult to reduce, ensuring the attack targets the most influential futures.

2. Dual Gating Mechanisms

To stabilize the optimization of the tail-aware loss and prevent the attack from degrading clean performance or becoming unstable, TRAP introduces two gating losses:

Sign Gate ( $L_{sign}$ ): Penalizes updates where the trigger inadvertently increases the score of a trajectory ( $\Delta J > 0$ ). This ensures the attack direction remains consistent (suppression).
Magnitude Gate ( $L_{mag}$ ): Prevents overly aggressive suppression. If a trajectory's score deviation exceeds a safety margin ( $\Delta J < -\beta$ ), further suppression is penalized. This maintains optimization stability and prevents unnecessary distortion.

The total attack objective combines the tail-aware loss, the dual gating losses, and standard image-space regularizers (Total Variation and $L_2$ norm) to ensure the trigger patch is spatially smooth and bounded.

Key Contributions

First Inference-Time Backdoor for World Models: TRAP is the first framework to perform trigger-conditioned backdoor attacks on world models during inference, manipulating long-horizon decision-making via visual triggers without requiring parameter tampering.
Identification of Ranking Vulnerability: The work identifies the relative ranking of candidate trajectories as the primary attack surface in world-model planning, moving beyond conventional objectives like one-step prediction errors or latent representation manipulation.
Tail-Aware Attack Formulation: The authors introduce a novel tail-aware ranking objective combined with dual gating mechanisms to achieve stable, behavior-preserving manipulation of planning outcomes.
Empirical Validation: The framework is validated on representative world models (DreamerV3 and TD-MPC2) across diverse tasks, demonstrating high effectiveness and transferability.

Experimental Results

The authors evaluated TRAP on DreamerV3 (latent imagination-based policy optimization) and TD-MPC2 (model predictive control) across continuous control (DMControl), sparse-reward planning (Crafter), and discrete action (Atari) environments.

Attack Effectiveness: TRAP consistently outperforms random patch baselines.
- On DreamerV3 (Crafter), TRAP achieved a 63.2% mean return drop and 98.1% Attack Success Rate (ASR), compared to negligible degradation from the baseline.
- On TD-MPC2 (hopper-hop), TRAP achieved a 99.8% mean return drop and 100% ASR.
- Across most tasks, TRAP drove ASR to near 100%, indicating persistent behavioral deviation whenever the trigger was present.
Clean Fidelity: The attack maintains normal behavior on clean inputs, with minimal impact on performance when the trigger is absent.
Efficiency: The attack incurs only modest computational overhead (runtime overhead generally between 0.8x and 1.3x relative to clean evaluation).
Robustness: Diagnostic experiments show TRAP remains effective across varying patch sizes and perturbation budgets. It also demonstrates resilience against standard image-space defenses (e.g., median filtering, JPEG compression, Gaussian smoothing), though these defenses do reduce effectiveness to some degree.

Significance and Claims

The paper claims that TRAP exposes a fundamental security gap in world-model-based agents. The significance of the work lies in demonstrating that:

Structural Vulnerability: The reliance on ranking high-value tail trajectories for decision-making creates a specific vulnerability that standard backdoor attacks miss.
Persistence of Attack: Unlike attacks on reactive models, TRAP induces sustained behavioral deviations over long horizons because it disrupts the trajectory ordering that the planner relies on for future evaluation.
Need for New Defenses: The findings highlight that current security evaluations are insufficient for world-model-driven agents. The authors argue for the necessity of dedicated security evaluation, robustness analysis, and defense design specifically tailored to the planning mechanisms of world models before their deployment in safety-critical applications like robotics and autonomous driving.