💻 computer science

ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration

ActiveGlasses is a system that enables zero-shot transfer of robot manipulation skills from ego-centric human demonstrations by using a stereo camera on smart glasses to capture coordinated perception and action, which is then replicated on a robotic arm via an object-centric point-cloud policy.

Original authors: Yanwen Zou, Chenyang Shi, Wenye Yu, Han Xue, Jun Lv, Ye Pan, Chuan Wen, Cewu Lu

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanwen Zou, Chenyang Shi, Wenye Yu, Han Xue, Jun Lv, Ye Pan, Chuan Wen, Cewu Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Robots are "Data Hungry" but "Clumsy Learners"

Imagine you want to teach a robot how to do chores, like putting a book on a shelf or pouring water without spilling. To do this, the robot needs to watch humans do it first (a process called "Imitation Learning").

But here's the catch:

The "Heavy Backpack" Problem: Current ways to teach robots are exhausting. Operators often have to wear bulky VR headsets or hold heavy controllers that mimic robot arms. It's like trying to learn to dance while wearing a 50-pound backpack. It's tiring, slow, and the data you get isn't very natural.
The "Passive Eye" Problem: Most robots have cameras stuck on their wrists. This is like having a camera glued to your hand. If you want to see around a corner, your hand has to move there first. But humans don't work that way! We move our heads to peek around obstacles before we move our hands. Robots with wrist-cameras miss this "active vision" and often get stuck because they can't see the task clearly.

The Solution: "ActiveGlasses"

The researchers built a system called ActiveGlasses. Think of it as giving the robot a pair of "smart human eyes" and a "human brain" without forcing it to wear a heavy backpack.

Here is how it works, broken down into three simple steps:

1. The "Bare-Hand" Teacher (Data Collection)

Instead of making a human operator wear a heavy robot controller, they just wear a pair of smart glasses (like XREAL Air 2) with a small stereo camera attached.

The Analogy: Imagine you are teaching a robot to bake a cake. Instead of making the teacher wear a giant, clunky suit that forces them to move like a robot, you just let them wear normal glasses and use their own hands. They move naturally, peek around the oven, and grab ingredients exactly how a human would.
The Result: The system records exactly what the human sees (stereo video) and exactly how they move their head (6-DoF tracking). This captures the "active vision"—the natural way humans look around to solve problems.

2. The "Object-Focused" Brain (The Policy)

This is the clever part. Usually, if you teach a robot using human hands, the robot gets confused because it has metal claws, not fingers.

The Analogy: Imagine you are teaching a dog to fetch a ball. You don't teach the dog how to move its paws to look like a human; you teach it where the ball is going.
How it works: The AI doesn't try to copy the human's hand movements. Instead, it looks at the 3D world (like a point cloud) and predicts where the object (the book, the bread, the teapot) needs to go. It ignores the "how" (fingers vs. claws) and focuses on the "what" (moving the object from A to B). This allows the robot to learn from a human and then immediately use its own robot arms to do the same job, even if the arms look totally different.

3. The "Head-Mimicking" Robot (Deployment)

When the robot goes to work, it doesn't just sit there. It has a special setup:

Arm A (The Worker): This arm grabs the object and moves it based on the AI's prediction.
Arm B (The Head): This is a second robotic arm holding the camera. It mimics the human's head movements recorded earlier.
The Analogy: If the human teacher leaned in to peek under a table to see a lost coin, the robot's "Head Arm" will also lean in to peek. This allows the robot to see around obstacles and solve tricky tasks that a fixed camera would miss.

What Did They Test?

They tried this on three tricky tasks that usually stump robots:

Book Placement: Putting a book on a shelf where the view is blocked by the side of the shelf.
Bread Insertion: Sliding bread into a toaster where the slot is hidden until you look at it from the right angle.
Water Pouring: Pouring water into a cup that is hidden behind a screen.

The Results

Zero-Shot Transfer: The robot learned from the human wearing glasses and then immediately tried the task on a real robot without any extra training. It just worked.
Beating the Baselines: In tests, ActiveGlasses was 25% to 35% more successful than other advanced methods.
Why? Because it combined natural human movement (no heavy backpacks) with active vision (moving the camera to see better) and a smart brain that focuses on the object, not the specific body parts.

The Takeaway

ActiveGlasses is like a universal translator for robots. It translates the natural, intuitive way humans interact with the world (moving our heads to see, using our hands freely) into instructions that robots can understand and execute, even if they look nothing like us. It solves the "data hunger" crisis by making data collection fast, comfortable, and incredibly effective.

1. Problem Statement

The paper addresses two critical bottlenecks in scaling robot learning from human demonstrations:

Data Collection Inefficiency & Burden: Existing pipelines often rely on teleoperation or bulky handheld devices (e.g., force-feedback controllers, heavy VR headsets). These methods force operators to mimic robotic kinematics rather than using their natural bare-hand movements, leading to physical exhaustion, suboptimal data quality, and limited scalability.
The Embodiment & Perception Gap:
- Manipulation: Mapping human hands to robot grippers is error-prone.
- Perception: Most systems use fixed third-person cameras or passive wrist-mounted cameras. Humans, however, utilize active vision—instinctively moving their heads to peer around obstacles, lean in for precision, and adjust viewpoints independently of hand movements. Current systems fail to capture this "intent-driven" perceptual signal, leading to failures in occluded or precise tasks.

2. Methodology: The ActiveGlasses System

The authors propose ActiveGlasses, a lightweight, head-mounted system that captures natural human manipulation and active vision, coupled with an object-centric policy for zero-shot transfer to robots.

A. Hardware & Data Collection

Setup: Operators wear XREAL Air 2 Ultra smart glasses equipped with a ZED Mini stereo camera.
Interaction: Operators perform tasks using bare hands (no handheld devices).
Sensors:
- Vision: The ZED Mini captures stereo RGB-D video.
- Motion: The XREAL's internal 6-DoF tracking module records the operator's head trajectory (simulating neck/torso movement).
- Interface: A Unity-based UI on the glasses provides audio/gesture feedback for episode start/stop.
Data Processing:
- Depth & Point Clouds: Stereo images are processed via FoundationStereo to generate depth maps and 3D point clouds.
- Segmentation: Grounded-SAM removes hands; SAM2 segments the target object.
- Calibration: A robust calibration method using three fixed spheres on the table establishes a world frame, propagating transforms via head-pose relative motion to handle occlusions of the calibration markers.

B. Policy Design: Object-Centric 3D Policy

To bridge the gap between human hands/heads and robot arms, the system avoids direct kinematic retargeting. Instead, it predicts object trajectories in 3D space.

Input: Unified 3D point clouds (in the world frame) derived from the stereo camera and head pose.
Architecture: A 3D Transformer backbone with two separate Diffusion Heads:
1. Manipulation Head: Predicts the absolute 6-DoF trajectory of the target object.
2. Perception Head: Predicts the relative head movement trajectory (to mimic active vision).
Key Design Choices:
- Absolute vs. Relative: Object trajectories are predicted in absolute coordinates to stabilize learning; head movements are relative to avoid workspace limits.
- No Current Pose Conditioning: The policy does not take the current object pose as an explicit condition. This forces the model to rely on visual reasoning rather than memorizing fixed trajectories based on initial state, improving generalization.
Execution:
- Manipulation Arm: Executes the predicted object trajectory.
- Perception Arm: A secondary 6-DoF arm (mounted on a table) dynamically mimics the predicted human head movements, providing the robot with active vision capabilities.

3. Key Contributions

Scalable Bare-Hand Data Collection: A system that eliminates the need for teleoperation or heavy handheld devices, allowing operators to collect data naturally with bare hands while wearing lightweight glasses.
Active Vision-Only Input: The framework relies solely on a single active-vision camera (mounted on the robot's "head" arm) for both training and inference. It successfully handles occlusions and distant interactions without fixed external cameras or wrist-mounted sensors.
Object-Centric Cross-Embodiment Policy: By predicting object trajectories in 3D space rather than robot joint angles, the system achieves zero-shot transfer across different robot platforms (e.g., Flexiv Rizon4 and UR5) without retraining or retargeting.
First-of-its-Kind Integration: It is the first work to combine smart glasses for active vision data collection with an object-centric 3D policy for zero-shot robot deployment.

4. Experimental Results

The system was evaluated on three challenging real-world tasks involving occlusion and precision:

Book Placement: Inserting a book into a partially occluded shelf.
Occluded Distant Water Pouring: Pouring water into a cup hidden behind a screen.
Bread Insertion: Inserting bread into a toaster slot not initially visible.

Performance Metrics:

vs. Baselines: Under the same single-camera hardware setup, ActiveGlasses outperformed the strong baseline $\pi_0.5$ (trained on real-robot teleoperation data) by significant margins in final success rates:
- Book Placement: +35%
- Bread Insertion: +25%
- Water Pouring: +30%
Ablation Studies:
- Active Vision: Removing active vision (fixed camera) caused a drastic drop in performance, especially in occluded tasks (e.g., Bread Insertion success dropped from 15/20 to 10/20).
- Policy Design: Using absolute object trajectory representation yielded better results than relative. Conditioning on the current object pose degraded performance, as the model tended to ignore visual inputs.
Cross-Embodiment: The policy successfully transferred from a Flexiv Rizon4 to a UR5 robot with comparable performance in early stages, demonstrating robustness to different kinematic structures.

5. Significance

Paradigm Shift in Data Collection: ActiveGlasses demonstrates that high-quality robot learning data can be collected efficiently by humans acting naturally, removing the "embodiment gap" during the data acquisition phase.
Active Vision as a Learnable Signal: The paper proves that head movement is not just noise but a critical, learnable signal for solving occlusion and precision tasks. By explicitly modeling and reproducing this behavior, robots can achieve human-like situational awareness.
Scalability: The combination of low-cost hardware (smart glasses + stereo camera), natural interaction (bare hands), and object-centric policies offers a highly scalable path toward deploying general-purpose robots in everyday environments.