Humanoid Hanoi: Investigating Shared Whole-Body Control for Skill-Based Box Rearrangement

1. Problem Statement

The paper addresses the challenge of long-horizon humanoid box rearrangement, specifically the task of transforming an initial configuration of stacked boxes into a target configuration (similar to the Tower of Hanoi puzzle) under strict placement and stacking constraints.

Key Challenges:

• Error Accumulation: Long-horizon execution compounds small errors in locomotion and manipulation, leading to "off-nominal" states (e.g., slightly tilted boxes or misaligned robot poses) that isolated skill evaluations often fail to detect.
• Distribution Shift: Naively composing independently trained skills induces new state and command distributions that differ from the training data of individual skills, causing robustness degradation.
• Controller Switching: Traditional approaches often switch between separate low-level controllers for different skills, which can cause transient instability at skill boundaries and fail to exploit shared whole-body dynamics.

2. Methodology

The authors propose a modular skill-based control framework centered around a Shared, Task-Agnostic Whole-Body Controller (WBC).

A. Architecture Overview

• High-Level Skills: The system uses a library of reusable, independently trained skills:
  • GoTo: Locomotion to a target pose (unloaded and carrying a box).
  • Pickup: Grasping boxes of varying sizes/masses.
  • Place: Placing boxes on the floor or other boxes (stacking).
• Shared WBC: Instead of switching controllers, all skills route their high-level commands through a single, always-on Masked Humanoid Controller (MHC).
  • The MHC is a learned policy trained to execute partially specified motion directives (masked pose components).
  • Skills output specific constraints (e.g., arm targets, base velocity), while the WBC autonomously fills in the remaining degrees of freedom (e.g., balancing, leg coordination) in a dynamically consistent manner.
  • This ensures a unified closed-loop control structure, avoiding the instability of switching low-level laws.

B. Shared WBC Coverage Expansion

The authors identify that a pretrained WBC may lack coverage for the specific state/command distributions induced by composed skill execution. To address this, they introduce a Rollout-Based Data Aggregation procedure:

1. Execution: Run the hierarchical system (Skills + Shared WBC) in simulation to generate successful closed-loop rollouts.
2. Aggregation: Extract the sequence of masked motion directives (targets and masks) issued to the WBC during these successful rollouts.
3. Retraining: Augment the WBC's training dataset with these new trajectories, applying domain randomization (mass, friction, delays) and external disturbances.
4. Optimization: Continue training the shared WBC on this expanded dataset using the original task-agnostic objective, rather than skill-specific rewards. This expands the controller's robustness to "corner cases" without introducing skill-specific parameters.

C. The Humanoid Hanoi Benchmark

To evaluate the approach, the authors introduce Humanoid Hanoi, a benchmark requiring the robot to move and stack three boxes across three towers.

• Constraints: Only one box moved at a time; larger boxes cannot be placed on smaller ones.
• Stress Test: The task forces repeated skill chaining over ~7 moves (28 skill invocations), exposing the system to accumulated errors and diverse geometric configurations.

3. Key Contributions

1. Shared-WBC Skill Composition: A novel architecture where independently trained skills are composed at the task level but executed through a single, fixed low-level controller, simplifying composition and improving stability.
2. Coverage Expansion via Data Aggregation: A method to maintain and improve the shared WBC's robustness by aggregating closed-loop execution data, effectively handling distribution shifts caused by skill composition without per-skill fine-tuning.
3. Humanoid Hanoi Benchmark: A publicly available benchmark for long-horizon box rearrangement that quantifies task success and precision, specifically designed to expose failure modes in extended autonomy.

4. Results

Simulation Experiments

• Individual Skills: The proposed Extended method (Shared WBC + Data Aggregation) outperformed baselines (Frozen MHC, Per-Skill Finetuning, and Residual Policies) in Pickup and Place tasks, particularly for low-height boxes (<0.2m) where balance margins are tight. It achieved the highest success rates and lowest Center of Mass (CoM) errors.
• Long-Horizon Performance (Humanoid Hanoi):
  • Without Domain Randomization: The Extended method achieved ~70% complete-task success, significantly outperforming the "Finetune" baseline, which degraded sharply after the first few moves.
  • With Domain Randomization: Extended maintained a 49% success rate, a substantial margin over other methods, demonstrating superior robustness to environmental variability.
  • Failure Analysis: Most failures occurred during the Place skill due to accumulated upstream errors (grasp misalignment, localization drift). The Extended method was more likely to stabilize and complete the task even with non-ideal poses, whereas baselines failed earlier.

Hardware Experiments (Digit V3)

• The system was deployed on the Digit V3 humanoid robot.
• Performance: Achieved a 40% success rate (6/15 trials) across three fixed configurations, with successful trials taking approximately 5 minutes.
• Failure Modes: Hardware failures were primarily attributed to:
  • Perception issues (AprilTag detection failures).
  • Navigation/alignment errors at skill handoffs.
  • Physical interactions (box flipping, collisions during recovery).
• Despite hardware limitations, the system demonstrated fully autonomous pickup, transport, and stacking.

5. Significance

This work demonstrates that long-horizon autonomy for complex humanoid tasks is feasible without resorting to brittle, skill-specific low-level controllers. By treating the whole-body controller as a maintained, shared component that is iteratively refined via data aggregation, the system achieves:

• Scalability: New skills can be added without retraining the entire control stack or switching control laws.
• Robustness: The system handles the distribution shifts inherent in multi-step tasks better than methods that optimize skills in isolation.
• Practicality: The approach bridges the sim-to-real gap, enabling complex rearrangement tasks on real hardware with minimal architectural changes.

The paper suggests that future improvements should focus on object-aware locomotion (to prevent collisions with stacked boxes) and onboard perception (to replace external markers like AprilTags) to further enhance reliability.