MRS: Multi-Resolution Skills for HRL Agents

1. Problem Statement

The paper addresses a critical performance gap in Hierarchical Reinforcement Learning (HRL), specifically in subgoal-based frameworks (e.g., the Director architecture). While HRL excels at long-horizon planning by decomposing policies into a Manager (proposing subgoals) and a Worker (executing primitive actions), it consistently underperforms monolithic agents on agility-demanding tasks (e.g., dense-reward locomotion).

The authors identify two root causes for this failure:

1. Unconstrained Goal Representations: In standard HRL, the Manager learns a goal space without constraints on reachability or temporal distance. It can propose any state as a subgoal, even if the Worker cannot reach it. This dilutes representational capacity across the entire state manifold, making precise local control difficult.
2. The Precision-Smoothness Tradeoff: The optimal subgoal distance is neither fixed nor task-invariant.
   • Short subgoals: Enable precise local control on curves but amplify prediction noise, leading to jitter.
   • Long subgoals: Produce smooth trajectories on straight paths but accumulate geometric errors at sharp turns (cutting corners).
   • The Gap: No single fixed temporal resolution is universally optimal; the ideal distance depends on both the specific task and the current state within the task.

2. Methodology: Multi-Resolution Skills (MRS)

The authors propose Multi-Resolution Skills (MRS), a framework that learns multiple goal-prediction modules, each specialized for a fixed temporal horizon, and dynamically selects among them.

Core Architecture

• Conditional VAEs (CVAEs): Instead of a single unconditional autoencoder, MRS trains $N$ CVAEs in parallel. Each CVAE $i$ is specialized to predict states reachable exactly $l_i$ steps in the future ($s_{t+l_i}$) given the current state $s_t$.
  • Encoder: Maps the transition $(s_t, s_{t+l_i})$ to a latent skill $z$.
  • Decoder: Reconstructs the target state conditioned on $s_t$ and $z$.
  • Shared Backbone: To minimize parameter overhead, all CVAEs share a common backbone network, with only resolution-specific encoder/decoder heads.
• Resolution Set: The temporal horizons are geometrically spaced: $l \in \{K, 2K, 4K, 8K, \infty\}$, where $K$ is the manager's refresh interval. The $\infty$-skill is an unconditional VAE that acts as a robustness mechanism during early training.
• Meta-Controller: A learned policy component within the Manager that:
  1. Samples a latent skill $z_i$ from each resolution-specific head.
  2. Decodes these into candidate subgoals.
  3. Uses a choice head to output a one-hot distribution over resolutions, selecting the most appropriate skill based on the current state $s_t$.
• Training: All components (CVAEs, Manager, Worker) are trained jointly in a single end-to-end phase using the Recurrent State Space Model (RSSM) for imagination.
  • Exploratory Reward: The Manager is rewarded for executing transitions that are poorly reconstructed by the best-fitting finite CVAE (minimum reconstruction error across modules). This encourages the discovery of diverse state transitions across all temporal scales.

3. Key Contributions

1. Characterization of the HRL Agility Gap: The paper formally identifies that the performance gap between HRL and monolithic agents on agile tasks stems from unconstrained goal representations and the inability to adapt subgoal distance to local state geometry (precision vs. smoothness tradeoff).
2. MRS Framework: Introduction of a novel architecture that learns multiple goal-prediction modules with fixed temporal horizons and a meta-controller for state-dependent selection. This imposes a structural inductive bias (temporal partitioning) without increasing parameter count significantly.
3. Empirical Validation: Demonstration that MRS consistently outperforms fixed-resolution baselines and significantly narrows the performance gap with state-of-the-art non-HRL agents (e.g., DreamerV3) on agility benchmarks, while maintaining superiority on long-horizon sparse-reward tasks.

4. Experimental Results

The method was evaluated on three benchmark families: DeepMind Control Suite (DMC), Gym-Robotics, and AntMaze.

• Agility Tasks (DMC): MRS significantly outperformed the Director baseline (the unconstrained HRL state-of-the-art) on tasks like cheetah_run, quadruped_run, and hopper_hop. It reduced the performance gap with DreamerV3 (the non-HRL SOTA), achieving competitive or superior scores in most locomotion tasks.
• Sparse Reward & Long-Horizon (AntMaze, Gym-Robotics): MRS matched or exceeded Director on sparse-reward manipulation and long-horizon navigation, proving that adding temporal structure does not degrade long-horizon planning capabilities.
• Ablation Studies:
  • Single-Resolution: Agents trained with only one fixed skill length ($l=16, 32, 64$) or the $\infty$-skill (Director) consistently underperformed the full MRS, confirming that the gains come from the multi-resolution capability, not just the presence of a specific skill length.
  • Meta-Controller: Replacing the learned selection mechanism with random selection or forcing a single skill resulted in lower performance, proving the value of state-dependent adaptive selection.
  • Exploration: The specific MRS exploratory objective (min-reconstruction error) yielded faster convergence and better fine-tuning performance compared to vanilla exploration objectives.
• Qualitative Analysis: t-SNE visualizations showed that the meta-controller learns a structured, state-dependent policy, selecting short skills for high-curvature regions (turns) and long skills for straight paths.

5. Significance and Implications

• Structural Inductive Bias: The paper demonstrates that the HRL agility gap is primarily a representational problem, not a capacity or training frequency issue. Imposing a temporal structure on the goal space is sufficient to recover precision lost in standard HRL.
• Efficiency: MRS achieves these gains with minimal parameter overhead (shared backbone) and without requiring task-specific tuning or privileged state information.
• Generalizability: The approach suggests that other subgoal-based HRL architectures could benefit from similar temporal partitioning. It bridges the gap between unsupervised skill discovery (diversity) and goal-conditioned HRL (temporal structure), offering a robust solution for agents that must handle both long-horizon planning and fine-grained, reactive control.