Technical Summary: Hierarchical Post-Training for Embodied Agents (HiRoC)

Problem Statement

Vision-Language-Action (VLA) models have shown promise in robotic manipulation by unifying visual perception, language understanding, and action generation. However, existing post-training methods predominantly treat VLA models as flat policies, mapping global task instructions directly to low-level actions. This approach faces significant limitations in real-world scenarios:

1. Lack of Explicit Task Progression: Flat policies maintain the same global instruction throughout execution, failing to explicitly model the stage-wise structure of complex, long-horizon tasks.
2. Accumulation of Errors: Without intermediate semantic anchors (subgoals), minor execution deviations accumulate over time, leading to degraded decision-making or task failure.
3. Ineffective Hierarchical Learning: While hierarchical approaches exist, they often rely on supervised learning from offline demonstrations and struggle to improve execution through online interaction. Furthermore, existing online Reinforcement Learning (RL) methods still optimize flat policies rather than hierarchical planner-executor structures.

The core challenge lies in creating a post-training framework that effectively decouples high-level planning from low-level execution while ensuring the two modules collaborate efficiently without suffering from distribution misalignment.

Methodology: HiRoC Framework

The authors propose Hierarchical Robotic Control (HiRoC), a hierarchical post-training framework that integrates task decomposition, planner-executor alignment, and online reinforcement learning. The framework consists of two primary modules: a high-level Planner and a low-level Executor.

1. Training the Planner via Supervised Fine-Tuning (SFT)

The planner is responsible for decomposing global task instructions into executable intermediate subgoals based on current observations.

• Data Preparation: The authors reorganize existing multimodal datasets (e.g., from VLA-OS) to create a clean, consistent dataset ($D_{plan}$) where trajectories are annotated with immediate next subgoals.
• Optimization: The planner (based on a VLM backbone like Qwen2.5-VL) is trained via SFT to predict the next subgoal token sequence conditioned on the current multimodal state ($s_t$) and previous subgoal tokens.
• Freezing: Once trained, the planner is frozen during subsequent RL tuning to preserve its task decomposition capabilities and prevent distribution shifts.

2. Training the Executor via SFT and RL Tuning

The executor generates low-level actions conditioned on the current subgoal provided by the planner.

• Addressing Distribution Misalignment (DM): A critical challenge is that pre-trained VLAs are optimized for global goals, not intermediate subgoals. Directly coupling a planner with a pre-trained executor leads to a "cold start" due to distribution misalignment.
  • Solution: The authors reorganize trajectory data into subgoal-conditioned samples ($s_i, l_i, a_i$) and perform Subgoal-Conditioned SFT on the executor. This aligns the executor's policy with the planner's output before RL begins.
• Online Reinforcement Learning (RL): The executor is further optimized via online interaction with the environment, guided by the frozen planner.
  • Hierarchical GRPO: The authors introduce a Group Relative Policy Optimization (GRPO) objective that incorporates both global and local learning signals.
    • Global Advantage ($A_{task}$): Derived from the cumulative reward of the full trajectory.
    • Local Advantage ($A_{sub}$): Derived from the progress made on intermediate subgoals recorded during rollout.
  • Weighted Integration: The final advantage signal ($\hat{A}_i$) is a weighted sum of global and local advantages. The weights ($w_t, w_s$) are adaptively balanced based on the average number of subgoals per trajectory and trajectory length, ensuring that tasks with varying complexity receive appropriate supervision.
  • Policy Update: The executor is updated using a PPO-style clipped actor objective, optimizing the probability ratio of action tokens while limiting policy deviation.

Key Contributions

1. HiRoC Framework: The proposal of a hierarchical post-training framework that explicitly separates high-level task planning from low-level action execution, enabling structured reasoning and robust long-horizon manipulation.
2. Collaborative Training Strategy:
   • Distribution Alignment: A subgoal-conditioned SFT stage that bridges the gap between planner-generated subgoals and executor execution, mitigating the cold-start problem.
   • Hierarchical RL Objective: A novel GRPO formulation that jointly exploits task-level (global) and subgoal-level (local) learning signals, providing more structured guidance for policy optimization.
3. Empirical Validation: Extensive experiments demonstrating consistent performance improvements across diverse robotic manipulation benchmarks.

Experimental Results

The authors evaluated HiRoC on the LIBERO benchmark (Goal, Spatial, Object, and Long suites) and the LIBERO-Plus generalization suite.

• Overall Performance: HiRoC achieved State-of-the-Art (SOTA) performance, outperforming strong baselines including flat VLA policies (OpenVLA), other RL-based methods (VLA-RL, TGRPO), and world-model-based approaches.
  • On the LIBERO-Long suite, HiRoC achieved a 98% success rate, significantly outperforming baselines and demonstrating the efficacy of hierarchical control for complex tasks.
  • The method showed an average improvement of 10.06% across benchmarks compared to baselines.
• Generalization (Zero-Shot): On LIBERO-Plus, which introduces seven types of perturbations, HiRoC demonstrated superior generalization compared to SFT-based models and world-model approaches, attributed to its ability to decompose goals into manageable subgoals.
• Ablation Studies:
  • Removing the planner (w/o Plan) significantly degraded performance, confirming the necessity of explicit task decomposition.
  • Removing the distribution misalignment (DM) pre-training resulted in a severe cold start and lower final performance.
  • Removing the local GRPO (w/o L) showed that while global GRPO is primary, the local subgoal signal provides complementary improvements.
• Real-World Transfer: A sim-to-real experiment on a physical robot platform successfully deployed the trained policy without additional fine-tuning, validating the framework's practical applicability.

Significance and Claims

The paper claims that HiRoC addresses the fundamental limitation of flat VLA policies in handling long-horizon, multi-stage robotic tasks. By explicitly modeling task progression through a planner-executor hierarchy and solving the distribution misalignment problem via targeted SFT and hierarchical RL, HiRoC enables agents to perform robust, self-correcting manipulation.

The authors emphasize that while their current work focuses on hierarchical post-training, future directions include exploring end-to-end training and more robust decomposition mechanisms to handle uncertainty in physical systems. The work underscores the importance of combining hierarchical task decomposition with reinforcement learning-based policy optimization for advancing general-purpose robotic manipulation.