Omni-Manip: Beyond-FOV Large-Workspace Humanoid Manipulation with Omnidirectional 3D Perception

This paper presents Omni-Manip, an end-to-end LiDAR-driven visuomotor policy that leverages a Time-Aware Attention Pooling mechanism to process 360° panoramic point clouds, enabling humanoid robots to perform robust dexterous manipulation in large, cluttered workspaces without the need for frequent repositioning or reliance on narrow-field-of-view RGB-D cameras.

The Gist

Imagine you are trying to clean up a messy room, but you are wearing a pair of goggles that only let you see a tiny circle directly in front of your nose. If a toy is on the floor to your left, you can't see it. If a chair is behind you, you don't know it's there. To get the toy, you have to stop, spin your whole body around, look, and then try again. This is exactly the problem current humanoid robots face when they try to do tasks in messy, real-world environments.

This paper introduces Omni-Manip, a new way to teach robots how to "see" and move that solves this problem. Here is the breakdown using simple analogies:

1. The Problem: The "Tunnel Vision" Robot

Most robots today use cameras (like RGB-D sensors) that act like a flashlight in a dark room. They only light up what is directly in front of them.

• The Issue: If a robot needs to pick up a cup that is behind it, or if there is a chair to its side, the robot is "blind" to it.
• The Consequence: The robot has to constantly stop, shuffle its feet, turn its head, and re-orient itself just to find the object. This is slow, clumsy, and risky because it might bump into things while turning around.

2. The Solution: The "360° Super-Vision"

The researchers gave the robot a new set of eyes: a LiDAR sensor (a laser scanner) mounted on its head.

• The Analogy: Instead of a flashlight, imagine the robot is wearing night-vision goggles that see in a perfect 360-degree circle, like a security camera that sees everything around the building at once.
• The Benefit: The robot can instantly "see" a bottle on a table behind it, a chair to its left, and a door to its right, all at the same time. It doesn't need to turn its body to know what's around it.

3. The Brain: "Time-Aware Attention"

LiDAR data is a bit tricky. It's like a swarm of millions of tiny, invisible dots floating in the air. Sometimes the dots flicker or move slightly because the robot is breathing or the air is moving.

• The Innovation: The team created a special brain filter called Time-Aware Attention Pooling.
• The Analogy: Imagine you are trying to hear a friend in a noisy crowd. If you just listen to one split-second of sound, you might hear a cough or a car horn instead of your friend. But if you listen to a few seconds of sound and focus on the voice that stays consistent, you understand them perfectly.
• How it works: The robot looks at the "dots" (point clouds) from the last few moments, not just the current one. It smooths out the noise and focuses on the most important parts, creating a stable, clear picture of the world.

4. The Teacher: "The Teleoperation Suit"

To teach the robot this new skill, they needed a lot of practice data. But you can't just tell a robot "move your arm here" easily.

• The Setup: They built a system where a human operator wears a VR headset and holds controllers (like a Meta Quest 3).
• The Analogy: Think of it like a digital puppet show. The human moves their arms and body in the real world, and the robot mimics them perfectly in real-time. Because the human is wearing the VR headset, they have the same "360-degree view" as the robot's LiDAR. This allows the human to teach the robot how to coordinate its whole body (legs, waist, and arms) to reach things without bumping into furniture.

5. The Results: The "Smooth Operator"

The researchers tested this robot in a messy room with obstacles.

• Old Robots (The Flashlight): They kept missing objects behind them or crashing into chairs because they couldn't see them. They failed almost every time the task required looking "out of view."
• Omni-Manip (The 360° Robot): It moved smoothly. It knew exactly where the chair was behind it, so it didn't bump into it. It reached for objects behind its back without ever turning its body. It was like a dancer who knows exactly where every person in the room is, even without turning their head.

Summary

Omni-Manip is like giving a robot a superpower: the ability to see everything around it at once, combined with a brain that remembers what it saw a second ago to stay steady. This allows humanoid robots to finally stop shuffling around clumsily and start working efficiently in the messy, unpredictable world we live in.

Technical Summary

Here is a detailed technical summary of the paper "Omni-Manip: Beyond-FOV Large-Workspace Humanoid Manipulation with Omnidirectional 3D Perception."

1. Problem Statement

The deployment of humanoid robots for dexterous manipulation in unstructured environments is currently hindered by perceptual limitations, specifically the narrow Field of View (FOV) of conventional RGB-D cameras.

• The Core Issue: Standard visuomotor policies rely on single-view or limited FOV inputs. In scenarios where the robot cannot easily reposition its base (due to physical constraints or soft ground), objects located outside the camera's view remain undetected.
• Consequences: This leads to task failures, inability to reach targets, and collisions with obstacles that are invisible to the robot's "eyes."
• Limitations of Existing Solutions:
  • Active Vision: Moving the camera/head adds mechanical complexity, latency, and control overhead.
  • Third-View Cameras: Require complex extrinsic calibration and struggle with generalization to new poses.
  • Standard 3D Point Clouds: Often derived from depth cameras, inheriting the same viewpoint constraints and self-occlusion issues.

2. Methodology: Omni-Manip

The authors propose Omni-Manip, an end-to-end, LiDAR-driven 3D visuomotor policy designed to enable robust manipulation across large workspaces using 360° omnidirectional perception.

A. System Architecture

The framework consists of four main components:

1. Hardware Platform: A Unitree G1 humanoid robot equipped with a Livox MID-360 LiDAR (mounted on the head) and Unitree Dex3-1 dexterous hands.
2. Data Collection: A custom Whole-Body Teleoperation System using Meta Quest 3 (HMD + controllers) to collect synchronized observation-action trajectories involving coordinated full-body movements.
3. Policy Learning (Omni-Manip):
   • Input: Panoramic point clouds (from LiDAR) + 43-dimensional proprioceptive joint states (full body).
   • Encoder: A Time-Aware Attention Pooling (TAP) mechanism.
     • Challenge: LiDAR point clouds are sparse and temporally flickering.
     • Solution: Aggregates point clouds over a short temporal window. Each point is augmented with a normalized relative timestamp. A small MLP predicts attention weights based on temporal offsets, allowing the model to emphasize recent observations and smooth out noise.
     • Processing: Raw LiDAR data is cropped to 1.3m (robot's reach) and downsampled to 4096 points per frame for efficiency.
   • Decoder: A Diffusion Policy that iteratively denoises random noise into coordinated action sequences (28-dimensional upper-body joint angles).
   • Deployment: The lower body and waist are controlled by a pre-trained locomotion algorithm (HOMIE), while the learned policy controls the upper body and hands.
4. Teleoperation Logic: Uses constrained inverse kinematics (solved via interior point optimization) to map XR controller poses to robot end-effectors, ensuring physical consistency and joint limits.

B. Key Technical Innovations

• Omnidirectional 3D Perception: Replaces RGB-D with panoramic LiDAR, providing 360° horizontal coverage. This eliminates "visual blind spots," allowing the robot to perceive and interact with objects anywhere around its body without repositioning.
• Time-Aware Attention Pooling (TAP): A novel encoding mechanism that handles the sparsity and temporal instability of LiDAR data by learning to weight points based on their temporal proximity, resulting in a stable global feature representation.
• End-to-End Learning: The system maps raw omnidirectional observations directly to whole-body actions without intermediate state estimation or explicit 3D reconstruction, reducing calibration dependencies.

3. Key Contributions

1. Omni-Manip Policy: An end-to-end LiDAR-driven visuomotor policy that enables large-workspace manipulation by processing panoramic point clouds with time-aware encoding.
2. Whole-Body Teleoperation System: A lightweight, XR-based system (using Meta Quest 3) for efficiently collecting high-quality human demonstration data involving complex full-body coordination.
3. Comprehensive Validation: Extensive experiments in both simulation (MuJoCo) and real-world environments demonstrating robust performance in large-workspace and cluttered scenarios where traditional vision-based methods fail.

4. Experimental Results

The authors evaluated Omni-Manip against three baselines: DP (Diffusion Policy on RGB), DP3 (Diffusion Policy on Depth-to-Point-Cloud), and iDP3 (Improved DP3).

• Task Success Rates:
  • Out-of-View (OV) Tasks: In tasks where targets were outside the camera's FOV (e.g., "Pour," "Hand Over," "Wipe"), baselines achieved 0% success. Omni-Manip achieved 100% success (12/12 to 16/20 depending on the specific task) in real-world OV scenarios.
  • In-View Tasks: Even in standard "Pick & Place" tasks, Omni-Manip outperformed baselines (e.g., 16/20 vs. 10-14/20 in simulation) due to better spatial reasoning and obstacle avoidance.
• Collision Avoidance:
  • In cluttered environments with obstacles placed outside the camera's view, baselines had collision rates of 90-100%.
  • Omni-Manip achieved a collision rate of only 25% (5/20) while maintaining a high success rate (14/20), proving its ability to leverage 360° data for safety.
• Generalization: Omni-Manip maintained high success rates across varying object instances, lighting conditions, and cluttered scenes, whereas baselines suffered significant performance drops.
• Ablation Study:
  • Removing panoramic LiDAR (using only egocentric depth) resulted in 0% success on large-workspace tasks.
  • Removing the Time-Aware Attention Pooling (TAP) reduced success rates, confirming the necessity of temporal smoothing for stability.

5. Significance and Impact

• Overcoming FOV Limitations: This work fundamentally shifts the paradigm for humanoid manipulation from "look-and-move" (repositioning to see) to "always-aware" manipulation. It proves that 360° perception is critical for operating in constrained, unstructured environments.
• Safety and Efficiency: By eliminating the need for frequent base repositioning and providing continuous obstacle awareness, the system reduces motion uncertainty and collision risks, which is vital for human-robot interaction.
• Scalability: The use of LiDAR (which is illumination-invariant and does not require complex extrinsic calibration for relative pose) offers a more robust and deployable solution compared to multi-camera setups or active vision systems.
• Future Direction: The paper establishes a strong foundation for integrating omnidirectional 3D perception into visuomotor learning, paving the way for fully autonomous humanoids capable of operating safely in complex real-world homes and workplaces.