💻 computer science

FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences

The paper introduces FetchMan, an end-to-end sim-to-real pipeline that overcomes the performance limitations of synthetic behavior cloning by refining policies with Flow-GRPO reinforcement learning, enabling a Unitree G1 humanoid to achieve 73.3% zero-shot success in loco-manipulation tasks across unseen scenes.

Original authors: Omar Rayyan, Zhi Li, Max Argus, Yuxin Jiang, Chang Yu, Chenfanfu Jiang, Yuchen Cui

Published 2026-09-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Omar Rayyan, Zhi Li, Max Argus, Yuxin Jiang, Chang Yu, Chenfanfu Jiang, Yuchen Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The dream of a robot that can walk into a room, spot a specific object, and pick it up has long been a central goal in robotics. For decades, researchers have struggled to teach machines this kind of complex behavior because it requires two difficult skills to work together at once: moving the body through space and using the hands to interact with the world. While robots have become quite good at walking on their own, adding the ability to reach and grasp things while balancing on two legs creates a chaotic mix of physics that is incredibly hard to program by hand. Collecting the necessary data to teach a robot this way is also a massive hurdle; having a human demonstrate every possible way to walk to a bowl and pick it up in every possible room would take years and risk breaking the robot. To solve this, scientists have turned to computer simulations, creating digital worlds where robots can practice millions of times without fear of damage. However, a major question has remained: can a robot trained entirely in a digital world actually learn to handle the messy, unpredictable reality of a real house?

A team of researchers at the University of California, Los Angeles, along with collaborators from the Allen Institute for AI and the University of Washington, has tackled this challenge with a new system they call FetchMan. Their work focuses on a humanoid robot, the Unitree G1, which looks and moves much like a person. The team wanted to know if they could teach this robot to navigate a room and grab a target object just by showing it thousands of examples in a computer simulation, and then have it perform the task perfectly in the real world without any further training. They discovered that simply showing the robot more and more examples of the task was not enough. Even after training on over 150,000 different scenes, the robot's performance hit a hard ceiling. It could mimic the digital teacher, but it could not learn the subtle timing required to transition smoothly from walking to reaching. The robot would often try to grab the object too early or stop walking too soon, failing because it was trying to copy a teacher that was using secret information the robot could not see.

To break through this barrier, the researchers added a second stage to the training process. Instead of just copying the digital teacher, they let the robot practice on its own in the simulation and rewarded it only when it successfully completed the entire task of walking to the object and picking it up. This method, which uses a technique called reinforcement learning, allowed the robot to figure out the correct timing and movement on its own. It learned to walk closer to the object before reaching out, correcting the mistakes it made when it was just imitating. When the team tested this refined robot in the real world, the results were striking. The robot, having never seen a real room or a real object during its training, was able to walk into unfamiliar spaces, identify a target bowl, and grasp it with a success rate of over 73 percent. This was a significant improvement over the initial training method, which only managed to succeed about 57 percent of the time in real-world tests.

The researchers also explored whether this approach could work for many different types of objects, not just bowls. They trained a version of the system to recognize and pick up items like bread, bottles, and books by giving the robot the name of the object as a clue. While this multi-object version was not quite as successful as the single-object version, it still managed to perform the task in a variety of situations, proving that the method can be scaled up. The study highlights that while copying a teacher is a good starting point, it is not enough to master complex physical tasks. The robot needs the freedom to explore and learn from its own successes and failures to truly understand how to move through the world. By combining a massive amount of simulated practice with a final stage of self-correction, the team has shown a clear path toward creating robots that can adapt to new environments without needing a human to hold their hand every step of the way.

Technical Summary: FetchMan

Problem Statement

The paper addresses the challenge of training visual humanoid loco-manipulation policies that can generalize to novel scenes and objects without per-scene tuning. While simulation has become standard for humanoid locomotion (walking, balancing), extending this to loco-manipulation (walking while grasping and moving objects) remains difficult. The primary hurdles are:

Data Scarcity: Collecting real-world data for humanoids is prohibitively expensive and risky due to the need for full-body balance recovery and safety supervision.
Simulation Limitations: Existing simulation approaches often rely on constrained visual environments or complex hand-crafted rewards.
The Imitation Gap: Training via Behavior Cloning (BC) on synthetic data often hits a performance ceiling. This is attributed to "privileged information" in scripted demonstrators (e.g., hidden phase variables) that the policy cannot observe, leading to discontinuous motion and an inability to learn the correct transition between navigation and manipulation phases.

Methodology

The authors propose FetchMan, an end-to-end sim-to-real pipeline trained entirely in simulation and deployed zero-shot on a real Unitree G1 humanoid. The approach consists of four main components:

1. Data Generation (FetchMan-Bench)

Scale: The pipeline generates over 150,000 episodes across procedurally generated indoor scenes (using MolmoSpaces) and diverse objects.
Scripted Demonstrator: A privileged whole-body controller ( $\pi_{ctrl}$ ) solves tasks using hidden state (e.g., object pose, occupancy grids, phase indices). It performs navigation via A* path planning and manipulation via a discrete phase sequence (reach, descend, close, lift).
Domain Randomization: To ensure visual generalization, every episode undergoes independent randomization of textures, lighting, camera intrinsics/extrinsics, surface heights, and action noise.
Episode Mixture: The dataset includes 80% "fetch" episodes (navigation + grasp) and 20% "pick-only" episodes (grasp only) to randomize initial upper-body configurations.

2. Policy Architecture

Observations: The policy ( $\pi_\theta$ ) receives a head-mounted fisheye RGB image, a wrist-mounted RGB image, and a proprioceptive vector (base height, roll/pitch, upper-body joint positions, gripper aperture).
Model: A Flow-Matching DiT (Diffusion Transformer) architecture.
- Vision: A frozen DINOv3 ViT backbone encodes visual inputs.
- Action Head: The transformer cross-attends to visual and state tokens to predict a chunk of 15-dimensional actions (base velocity, base height, waist/arm joint targets, gripper aperture).
- Control Interface: The policy outputs high-level commands tracked by a pre-trained SONIC lower-body controller (for balance/walking) and a PD controller (for upper body), reducing the action space from 29 joint torques to 15 meaningful commands.

3. Two-Stage Training Pipeline

The training addresses the imitation gap through a specific two-stage process:

Stage 1: Imitation Pretraining (Behavior Cloning): The policy is trained on the 150k synthetic demonstrations. This provides a strong prior for the high-dimensional control problem but is limited by the demonstrator's hidden phase structure.
Stage 2: Flow-GRPO Refinement: To break the performance ceiling, the policy is fine-tuned using Flow-GRPO (Group-Relative Policy Optimization adapted for flow-matching policies).
- Mechanism: Groups of environments are reset to identical tasks. The policy rolls out trajectories with injected noise. The advantage is calculated based on group-relative returns (success vs. failure) rather than a learned value function.
- Reward: A single sparse reward (1 for successful fetch, 0 otherwise) is used, avoiding per-step reward shaping.
- Outcome: This stage allows the policy to learn the implicit phase transitions (navigation-to-manipulation handoff) that the scripted demonstrator relied on hidden state to manage, effectively "repairing" the structural artifacts left by BC.

4. Multi-Object Extension

The recipe is extended to a text-conditioned variant for multi-object generalization. This model uses a dino.txt backbone to ground object names (e.g., "bowl," "pot") to visual tokens, enabling zero-shot grasping of diverse object categories.

Key Results

The paper evaluates FetchMan on FetchMan-Bench (a simulation benchmark) and deploys it on a real Unitree G1.

Imitation Ceiling: Scaling Behavior Cloning from 5k to 150k demonstrations improves success rate (SR) from 40% to 67% in simulation, but performance saturates at 67%, confirming that more data alone cannot overcome the privileged information gap.
RL Breakthrough: Applying Flow-GRPO to the 150k BC checkpoint lifts the simulation loco-manipulation SR to 83%.
Zero-Shot Real-World Performance:
- Single-Object: The final model achieves 73.3% loco-manipulation success on the real Unitree G1 across unseen scenes, with no real-world training data.
- Ablation: Replacing DINOv3 with SigLIP or using absolute action targets instead of delta targets causes real-world performance to collapse (dropping to 0% loco-manipulation SR), highlighting the necessity of geometric features and delta-action parameterization.
Multi-Object: The text-conditioned model achieves 62% SR in simulation (lower than the single-object 83% due to task complexity) and demonstrates successful zero-shot fetches of various objects (e.g., pots, bread, bottles) on the real robot, though with slightly less robustness than the single-object policy.

Significance and Claims

The paper claims to present the first end-to-end sim-to-real pipeline for visual humanoid loco-manipulation that generalizes across diverse scenes without per-environment tuning. Its significance lies in:

Demonstrating the Limits of Synthetic BC: It empirically shows that scaling synthetic demonstrations alone is insufficient for loco-manipulation due to the "imitation gap" caused by hidden phase variables in scripted controllers.
Validating Flow-GRPO for Humanoids: It proves that refining a cloned policy with group-relative RL on a sparse reward can break through the imitation ceiling, enabling the policy to learn complex phase transitions (walking to grasping) entirely in simulation.
Scalable Recipe: It provides a reproducible recipe (MolmoSpaces + Flow-GRPO) that scales from single-object to multi-object generalist policies, taking a "first step" toward task-generalist humanoids at this data scale.
Zero-Shot Deployment: It establishes that high-performance loco-manipulation can be achieved on real hardware using only simulation data, provided the training pipeline correctly addresses the structural limitations of imitation learning.

The authors remain modest, noting limitations such as the stateless nature of the policy (no history), the fixed lower-body controller (SONIC), and the current restriction to fetch tasks, leaving more complex manipulation and longer-horizon behaviors for future work.