Agent-Centric Visual Reinforcement Learning under Dynamic Perturbations

1. Problem Statement

Visual Reinforcement Learning (RL) agents learn policies directly from pixel observations but are highly vulnerable to dynamic visual perturbations (e.g., rain, snow, motion blur, sensor noise, and changing backgrounds). Existing benchmarks often rely on static background distractions or single fixed corruptions, failing to capture the non-stationary, time-varying nature of real-world environments where corruption types switch unpredictably (e.g., weather changing from rain to fog).

The paper identifies two critical failure modes in current methods:

1. Model-Free Methods: Encoders confuse task-relevant state features with corruption artifacts, leading to policy confusion when foreground textures are occluded.
2. Model-Based Methods (World Models): Reconstruction-based objectives (e.g., in DreamerV3) force the latent state to encode corruption-specific features to minimize reconstruction error. Under dynamic switching, this "entangles" the perturbation into the latent representation, contaminating the imagined rollouts used for planning and causing catastrophic performance drops.

Existing solutions like data augmentation or invariant representation learning fail because they either still feed corrupted pixels to the agent or cannot recover information lost at the pixel level (e.g., occluded foregrounds).

2. Methodology: ACO-MoE

The authors propose Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), a plug-and-play preprocessing framework designed to decouple perception from perturbation before the RL agent processes the input.

Core Components:

• Plug-and-Play Preprocessor: ACO-MoE sits between the environment and the RL backbone (encoder). It takes a corrupted observation $x_t$ and outputs a clean, agent-centric observation $\tilde{x}_t$.
• Mixture-of-Experts (MoE) Architecture:
  • Shared Encoder: Extracts multi-scale features from the corrupted input.
  • Router: A lightweight module that dynamically selects one of $|K|$ specialized experts based on the input features. Crucially, it operates without oracle labels of the corruption type during inference.
  • Restoration Experts: Each expert is specialized for a specific degradation type (e.g., rain, blur, noise).
• Dual-Stream Output: Each selected expert produces two outputs:
  1. RGB Residual: Restores the corrupted foreground pixels.
  2. Foreground Mask: Segments the agent and task-relevant objects from the background.
• Agent-Centric Composition: The final observation is constructed by compositing the restored foreground onto a uniform black background. This eliminates background distractions and ensures the RL agent only sees the task-relevant state.
• Training Strategy: The MoE is pretrained offline on paired clean/corrupted data with pseudo-masks. During RL training, the ACO-MoE module is frozen, preventing observation drift and ensuring stable policy learning.

3. Theoretical Analysis

The paper provides a formal information-theoretic analysis to justify the approach:

• Proposition 5.1 (Representation Contamination): It proves that for reconstruction-based world models, minimizing reconstruction error inevitably forces the latent representation $Z_t$ to encode the corruption mode $K_t$. Specifically, $I(Z_t; K_t | S_t) \ge I(X_t; K_t | S_t) - C(\epsilon)$. Accurate reconstruction implies high entanglement with noise.
• Corollary 5.2 (Foreground as IB Anchor): It demonstrates that extracting the clean foreground $F_t$ acts as an Information Bottleneck (IB) surrogate. Since the foreground is a deterministic function of the state $S_t$ and independent of the exogenous corruption $K_t$, using $F_t$ as input ($Z_t = F_t$) achieves exact nuisance invariance ($I(Z_t; K_t | Y_t) = 0$) while preserving task sufficiency.

4. Benchmark: Visual Degraded Control Suite (VDCS)

To systematically evaluate this problem, the authors introduce VDCS, an extension of the DeepMind Control Suite (DMC).

• Dynamic Perturbations: Unlike static benchmarks, VDCS uses Markov-switching dynamics to simulate non-stationary transitions between corruption types (e.g., rain $\to$ snow $\to$ blur) within a single episode.
• Degradations: Includes 7 physical degradations (Rain, Haze, Snow, Motion Blur, Gaussian Noise, Low-light, JPEG) and 2 background distractions (Video, Color).
• Severity: Corruptions have temporally correlated severity levels, mimicking real-world persistence.

5. Key Results

Experiments were conducted on VDCS and the DMControl Generalization Benchmark (DMC-GB) using both model-free (DrQ-v2) and model-based (DreamerV3, TD-MPC2) backbones.

• Performance on VDCS:
  • ACO-MoE achieves a 78.5% relative improvement over the best baseline (SimGRL) on average.
  • It recovers 95.3% of the clean performance of DreamerV3 under challenging Markov-switching corruptions.
  • Specific gains are massive on tasks sensitive to texture loss: finger_turn_hard (+207%), cheetah_run (+126%), and hopper_hop (+105%).
• Generalization (DMC-GB):
  • Achieves State-of-the-Art (SOTA) on DMC-GB (Video Hard and Color Hard), outperforming the second-best method by 6.2% and 17.4% respectively.
  • Demonstrates robustness even when the corruption type is not explicitly known at inference.
• Ablation Studies:
  • MoE vs. Single Expert: Specialized experts outperform a single large network, proving the benefit of routing.
  • Dual-Stream: Both RGB restoration and foreground masking are essential; removing either causes significant performance drops.
  • Frozen vs. Joint: Keeping the preprocessor frozen is critical; fine-tuning it during RL leads to instability and performance collapse.
• Efficiency: ACO-MoE adds minimal latency (~20ms/step on RTX 5090) compared to test-time adaptation methods like FTR, which rely on heavy segmentation models (SAM).

6. Significance and Contributions

1. New Benchmark (VDCS): Establishes a rigorous standard for evaluating visual RL under non-stationary, physically realistic perturbations, moving beyond static background distractions.
2. Theoretical Insight: Formally proves that reconstruction objectives in world models are structurally incompatible with robustness to dynamic corruptions, motivating the shift to agent-centric preprocessing.
3. Robust Framework (ACO-MoE): Introduces a novel, architecture-agnostic solution that decouples perception from perturbation. It works seamlessly with both model-free and model-based agents without requiring changes to the RL algorithm itself.
4. Practical Impact: The method recovers near-clean performance in highly degraded environments, making visual RL more viable for real-world deployment where sensor noise and environmental changes are inevitable.

In conclusion, this work shifts the paradigm from "learning to ignore noise" to "restoring the agent-centric view," providing a theoretically grounded and empirically superior solution for robust visual control.