Technical Summary: WMAttack

Problem Statement

Despite the increasing adoption of world models as decision-making agents in reinforcement learning (RL), their adversarial robustness remains underexplored. Current evaluation methods face a critical trade-off between accuracy and efficiency. Weak, manually tuned attacks often overestimate robustness, while exhaustive hyperparameter searches are computationally prohibitive. This is because evaluating a single attack candidate in a world-model setting requires closed-loop rollouts through learned latent dynamics, which is significantly more expensive than standard supervised learning evaluations. Furthermore, attack efficacy is highly sensitive to environment dynamics, observation spaces, and adversarial hyperparameters, making manual tuning unreliable. The paper identifies a lack of dedicated, automated frameworks to systematically expose and quantify the failure modes of world-model agents under adversarial conditions.

Methodology

The authors propose WMAttack, an automated attack-search framework designed to evaluate world-model agents under a finite evaluation budget. The core objective is formulated as a search problem over attack configurations (including attack families, perturbation budgets, optimization steps, restarts, and allocation rules) to maximize a scalarized utility function. This utility balances reward degradation, action instability, evaluation time, and rollout variability.

WMAttack employs a two-stage optimization process to navigate the large search space:

1. Representation-Guided Attack Retrieval (RGAR):
   To address the "cold-start" problem where evaluators lack prior knowledge of effective attacks for a new task-agent pair, RGAR retrieves historical attack configurations from representation-similar tasks. It constructs a latent behavioral summary ($\psi_\tau$) of the new task based on clean trajectories (latent states, predictions, actions, rewards). By projecting diverse dynamics into a unified latent space, RGAR identifies top-$K$ similar historical records and uses them to initialize the attack proposal distribution ($q_0$). This provides a warm start, leveraging cross-task experience rather than uninformed random initialization.

2. Self-Correcting Attack Search (SCAS):
   Following initialization, SCAS iteratively refines the search strategy using feedback from evaluated trials. It treats the ranked candidate generator as an implicit proposal distribution ($q_t$). After evaluating a batch of candidates via closed-loop rollouts, SCAS uses multi-dimensional feedback (reward degradation, action instability, runtime, variability) to update the proposal distribution. High-utility trials serve as anchors for local refinement, while feedback signals adjust search directions (e.g., modifying perturbation budgets or optimization steps). This self-correcting loop concentrates the evaluation budget on regions of the attack space more likely to expose world-model failures.

Theoretical Basis: The paper provides a theoretical explanation demonstrating that if the proposal refinement (via RGAR or SCAS) shifts probability mass toward high-utility attack configurations (the set $G_\eta$), the probability of discovering an effective attack within a finite budget increases. Specifically, the update rule $C_\gamma(q) = (q + \gamma q^*)/(1+\gamma)$, where $q^*$ is a utility-biased reference distribution, guarantees an increase in the probability of sampling an effective configuration if the reference is more concentrated on the effective set than the current proposal.

Key Contributions

1. Problem Formulation: The authors formulate adversarial robustness evaluation for world models as a finite-budget attack-configuration search problem, aiming to efficiently identify high-impact attacks across diverse tasks, models, and hyperparameter settings.
2. WMAttack Framework: They develop an automated framework combining RGAR and SCAS. RGAR enables cross-task experience transfer via latent behavioral representations, while SCAS utilizes feedback-driven proposal refinement to adaptively concentrate search efforts.
3. Theoretical Insight: The paper offers a theoretical explanation for why feedback-driven proposal refinement improves search efficiency, showing that shifting probability mass toward high-utility configurations reduces the expected hitting time for finding effective attacks.
4. Empirical Evaluation: Extensive experiments are conducted on 46 tasks across Atari and DeepMind Control (DMC) suites using various world-model architectures (DreamerV2, DreamerV3, TD-MPC2, IRIS).

Results

The evaluation demonstrates that WMAttack consistently discovers stronger attacks than both random search and a Claudini-style autoresearch baseline (which uses iterative feedback but lacks cross-task retrieval).

• Attack Strength: On DreamerV3 Atari, WMAttack achieved a normalized reward drop of 1.034, surpassing the Claudini baseline (0.497) and Random search (0.446) by significant margins. It was the only method to yield a positive scalarized utility (0.430), indicating the search cost was justified by the attack strength. On DMC, WMAttack improved the reward drop from 0.319 (Claudini) to 0.682.
• Cross-Model Generalization: The framework successfully exposed vulnerabilities in DreamerV2, TD-MPC2, and IRIS, achieving consistent gains in reward drop and action instability across these different architectures.
• Search Efficiency: WMAttack reached 90% of its final best utility in 87.5% of task-attack pairs on Atari, compared to ~53% for baselines. On DMC, the hit rate improved from ~18% to 49.0%.
• Ablation Studies:
  • RGAR vs. SCAS: RGAR significantly improved the initial candidate quality (first-round utility), while SCAS was essential for refining these candidates into high-utility final attacks. Neither component alone achieved the performance of the combined system.
  • LLM Backbones: The choice of LLM backbone (e.g., Qwen2.5-3B vs. 7B) affected performance non-monotonically. Smaller models (3B) sometimes outperformed larger ones (7B) in terms of reward drop and utility, suggesting that the effectiveness depends on the strength-cost trade-off rather than parameter count alone.

Significance and Claims

The paper claims that WMAttack bridges the gap between static adversarial evaluation protocols (like AutoAttack for supervised vision) and the dynamic, high-cost requirements of world-model evaluation. By treating evaluation as an automated finite-budget search, WMAttack provides a systematic way to expose failure modes that manual tuning or random search might miss.

The authors emphasize that their goal is not to certify global robustness but to efficiently discover high-impact attacks. The significance lies in the demonstration that leveraging intrinsic similarities between world models (via RGAR) and adapting search strategies based on closed-loop feedback (via SCAS) allows for the discovery of substantially more potent adversarial configurations under fixed computational budgets. This approach offers a practical pathway for the safety-oriented evaluation of increasingly complex world-model agents.