Moving Through Clutter: Scaling Data Collection and Benchmarking for 3D Scene-Aware Humanoid Locomotion via Virtual Reality

This paper introduces Moving Through Clutter (MTC), an open-source Virtual Reality framework that addresses the lack of data for scene-aware humanoid locomotion by procedurally generating diverse 3D cluttered environments, capturing whole-body human motion, and providing a benchmarked dataset of 348 trajectories to advance robot navigation in complex, real-world settings.

The Gist

Imagine you are teaching a robot to walk. So far, scientists have been very good at teaching robots to walk on empty, flat dance floors. They can make them run, jump, and even do parkour, but only if the room is completely empty.

But real life isn't an empty dance floor. Real life is a messy living room filled with coffee tables, piles of laundry, low-hanging lamps, and narrow hallways. If you put a robot in there, it would likely trip, crash into the sofa, or get stuck because it doesn't know how to squeeze through tight spots or duck under obstacles.

This paper introduces a new project called MTC (Moving Through Clutter) to solve exactly that problem. Here is how they did it, explained simply:

1. The Problem: The "Empty Room" Trap

Most robot training data comes from humans walking in big, empty studios. It's like teaching a driver to drive only in a parking lot with no other cars. When you finally put that driver on a busy city street with narrow alleys and parked cars, they panic. Robots are the same; they don't know how to adapt their bodies to squeeze through a cluttered house.

2. The Solution: The "Virtual Reality Simulator"

Instead of building hundreds of messy rooms in the real world (which would be expensive and take forever), the team built a Virtual Reality (VR) video game.

• The Game Master (Procedural Generation): They wrote a computer program that acts like a chaotic interior designer. It randomly builds rooms, filling them with furniture, debris, and obstacles. It can make a room slightly messy or extremely crowded, just like a real house.
• The Player (The Human): A human puts on a VR headset and walks through these digital, messy rooms. They have to dodge chairs, duck under beams, and squeeze through narrow gaps, just like they would in real life.
• The Magic Trick (Embodiment Scaling): This is the clever part. The robot they are training might be shorter or taller than the human. So, the VR system shrinks or stretches the virtual world to match the robot's size. If the robot is short, the human sees the room as "tall" and has to crouch more. This ensures the human's movements are perfectly sized for the robot.

3. The Result: A "Driving School" for Robots

The system records the human's movements in VR and translates them into data the robot can understand.

• The Dataset: They created a library of 348 different "walks" through 145 different messy rooms. It's like a massive textbook of "how to walk through a mess."
• The Test (The Benchmark): They didn't just collect data; they built a grading system. They check two things:
  1. Did you crash? (Collision Safety): Did the robot hit the furniture?
  2. Did you adapt? (Adaptation Score): Did the robot change its walk? (e.g., Did it lift its knees higher? Did it lean sideways? Did it crouch?)

4. Why This Matters

Think of this like giving a robot a Gym Membership.

• Before: Robots trained in a gym with no equipment. They were strong but clumsy in the real world.
• Now (With MTC): Robots are training in a gym filled with obstacles, ropes, and uneven floors. They are learning to twist, turn, and balance in tight spaces.

The paper shows that when robots learn from this new "messy room" data, they become much better at navigating real homes and offices without crashing. It moves robot walking from "perfect conditions" to "real-world chaos."

In a nutshell: The authors built a VR simulator where humans walk through messy digital rooms. They recorded how humans naturally adjust their bodies to avoid hitting things, scaled that data to fit a robot, and created a massive dataset to teach robots how to walk through clutter without falling over.

Technical Summary

Here is a detailed technical summary of the paper "Moving Through Clutter: Scaling Data Collection and Benchmarking for 3D Scene-Aware Humanoid Locomotion via Virtual Reality."

1. Problem Statement

Recent advances in humanoid locomotion have achieved impressive dynamic behaviors (e.g., parkour, dancing) but are predominantly trained and evaluated in open, flat, and obstacle-free environments. Real-world deployment requires navigating densely cluttered, 3D-constrained environments (homes, offices) where locomotion demands:

• Scene-aware whole-body coordination: Continuous adaptation of posture, lateral clearance, and limb placement.
• Precise balance control: Managing dynamic stability in irregular spaces.
• Geometric reasoning: Avoiding collisions with multi-height obstacles (furniture, debris).

Key Gaps Identified:

• Lack of Data: No public dataset systematically couples full-body human locomotion with the specific 3D scene geometry that shapes it. Existing datasets (e.g., AMASS) are recorded in open studios and lack environmental constraints.
• Scalability Bottleneck: Physically constructing diverse, cluttered environments for data collection is expensive and difficult to reproduce. Traditional motion capture fails to encode interactions with spatial constraints.
• Embodiment Mismatch: Human motion data often does not scale geometrically to specific robot morphologies, leading to infeasible postures or collisions when retargeted.

2. Methodology: The MTC Framework

The authors introduce Moving Through Clutter (MTC), an open-source framework comprising three core components: MTC Capturer, MTC Dataset, and MTC Benchmark.

A. MTC Capturer (Data Acquisition Pipeline)

This module enables scalable, embodiment-consistent data collection using Virtual Reality (VR).

• Procedural Environment Generation:
  • Generates 3D scenes with controllable clutter levels using two geometric regimes: Structured Domestic (furniture-based, corridor-like) and Debris-Style (irregular obstacles, overhead beams).
  • Hierarchical Layout: Assets are placed in layers (Anchor $\to$ Supporting Large $\to$ Small Clutter $\to$ Vertical Obstruction) to ensure semantic plausibility.
  • Clutterness Control: A scalar parameter $c \in [0, 1]$ controls floor-occupancy ratio.
  • Navigability Verification: Uses a 2D grid-based reachability test with morphology-aware inflation (expanding obstacles by the robot's clearance radius). If a path is blocked, an annealed resampling procedure iteratively removes expendable clutter (small items first) to restore connectivity while preserving global structure.
• Embodiment-Scaled Immersive Motion Capture:
  • Human operators navigate the virtual scenes using a PICO 4 Ultra VR system with full-body tracking (24 joints).
  • Scale Matching: To ensure geometric consistency, the virtual environment is rendered with a scale factor $\alpha = h_{robot} / h_{human}$. This ensures the human operator experiences spatial clearances identical to those the robot would encounter.
  • Retargeting: Captured human motion is rescaled and retargeted to the target humanoid model (Unitree G1) using a standard retargeting framework.

B. MTC Dataset

• Scale: Contains 348 trajectories across 145 diverse 3D scenes.
• Content: Approximately 731,000 motion frames (~2.3 hours of data).
• Variability: Scenes span domestic and debris regimes with realized floor-occupancy ratios ($c'$) ranging from 0.2 to 0.6.
• Diversity: Demonstrates that different goal placements within the same scene induce distinct traversal routes and behaviors (e.g., crouching, side-stepping, crawling).

C. MTC Benchmark

A quantitative evaluation framework to assess scene-aware locomotion performance.

1. Motion Adaptation Score: Measures deviation from nominal flat-ground walking across four subspaces:
   • Posture: Joint positions/velocities relative to the pelvis.
   • Vertical Motion: Pelvis height and acceleration (height clearance).
   • Foot Interaction: Foot heights and velocities (obstacle negotiation).
   • Smoothness: Jerk (third-order derivative) to measure dynamic modulation.
   • Metric: Uses Fréchet distance between the test trajectory's feature distribution and a baseline flat-ground distribution.
2. Collision Safety Assessment:
   • Evaluates trajectories against the original non-convex scene mesh using signed distance fields.
   • Metrics: Collision frequency ($R_{col}$), worst-case penetration depth ($d_{max}$), average penetration severity ($\bar{d}_{cond}$), and time-normalized penetration magnitude ($I_{pd}$).

3. Key Contributions

1. MTC Capturer: A novel VR-based paradigm that scales human motion to robot proportions, generating geometrically consistent trajectories in procedurally generated, cluttered environments without physical construction.
2. MTC Dataset: The first open-source dataset explicitly coupling whole-body locomotion trajectories with the 3D scene geometry that induces them, covering 145 unique scenes.
3. MTC Benchmark: A dual-evaluation protocol quantifying geometry-induced kinematic adaptation and collision safety, providing a standardized metric for scene-aware locomotion.
4. Procedural Generation Strategy: A robust pipeline for generating traversable, semantically structured cluttered scenes with guaranteed navigability via annealed resampling.

4. Results and Analysis

• Behavioral Diversity: Analysis of PCA projections on kinematic features shows that cluttered trajectories significantly deviate from the baseline (flat-ground) distribution in posture and foot interaction subspaces, confirming the dataset captures rich adaptation strategies.
• Smoothness: Despite geometric constraints, the collected human trajectories maintain high smoothness (low jerk), a property often lost in learned robot policies.
• Goal Conditioning: Case studies demonstrate that even within a single scene, different goals induce distinct behaviors (e.g., crouched lateral shuffling vs. prone crawling), proving the dataset captures goal-conditioned routing diversity.
• Preliminary RL Validation: A reinforcement learning policy trained to imitate MTC trajectories successfully reproduced geometry-induced traversal behaviors with low collision rates, validating the dataset's utility for training.

5. Significance and Future Work

• Significance: MTC bridges the gap between human motion priors and real-world deployment constraints. It shifts the focus from "balance on flat ground" to "navigating complex 3D geometry," providing the data foundation necessary for training scene-aware humanoid controllers.
• Limitations:
  • Current retargeting is scene-agnostic (does not explicitly learn collision avoidance from the scene geometry).
  • Scene generation relies on hand-crafted priors rather than learned generative models (e.g., VLMs).
  • The dataset focuses on locomotion, excluding contact-assisted progression (e.g., using hands for balance in extreme clutter).
  • VR-based capture may introduce pose estimation noise.
• Future Directions: Integrating high-precision motion capture, developing scene-aware generative models for environment creation, and extending the framework to include contact-assisted locomotion.

In summary, MTC provides a scalable, reproducible, and rigorous foundation for advancing humanoid robotics from controlled labs to the cluttered reality of human living spaces.