💻 computer science

Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation

Gaze2Act is a novel Vision-Language-Action framework that enhances robot manipulation by integrating human gaze as a dynamic intent signal, bridging ego-exo view gaps to achieve state-of-the-art performance in object disambiguation, fine-grained interaction, and dynamic intent steering on a Unitree G1 humanoid.

Original authors: Kuangji Zuo, Gen Li, Bofan Lyu, Yanshuo Lu, Boyu Ma, Shijia Han, Xinyu Zhou, Xichen Yuan, Chuhao Zhou, Jiaqi Bai, Geng Li, Jianfei Yang

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kuangji Zuo, Gen Li, Bofan Lyu, Yanshuo Lu, Boyu Ma, Shijia Han, Xinyu Zhou, Xichen Yuan, Chuhao Zhou, Jiaqi Bai, Geng Li, Jianfei Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to help you in the kitchen. You say, "Hand me that cup." But there are five cups on the table: one red, one blue, one with a crack, and two identical white ones. If you just say "the cup," the robot might grab the wrong one, or worse, grab the one you don't want.

This is the problem the paper Gaze2Act tries to solve. It argues that human language is often too vague for robots to understand exactly what we want, especially when we need to be precise or when our minds change mid-task.

Here is how the paper explains their solution, using simple analogies:

The Core Idea: "Eyes Lead the Hands"

The authors noticed something humans do naturally: we look at what we are about to touch before we touch it. If you want to pick up a specific apple, your eyes lock onto it a split second before your hand reaches out.

The paper proposes that instead of just talking to the robot, we should let the robot "see" where our eyes are looking. They call this Gaze2Act. It's like giving the robot a pair of "mind-reading glasses" that show it exactly what you are focusing on, in real-time.

How It Works: The Three-Step Magic Trick

The paper describes a system that bridges the gap between what you see and what the robot sees.

1. The Translator (Cross-View Grounding)

The Problem: You are wearing smart glasses looking at a cup from your perspective. The robot is looking at the same cup from a different angle. Your "gaze point" (where your eyes are looking) doesn't line up with the robot's camera view.
The Solution: The system acts like a super-smart translator. It takes your gaze point and uses a "visual matching" tool to find the exact same object in the robot's camera view. It draws a digital mask (like a highlighter) around the object you are looking at and marks the exact spot you are staring at.
Analogy: Imagine you are pointing at a specific tree in a forest from a hill. The robot is at the bottom of the hill. The system instantly draws a glowing circle around that exact tree in the robot's view, even though the angles are totally different.

2. The Highlighter (Perception-Level Prompting)

The Problem: Just knowing "where" isn't enough; the robot needs to know what to do with that information.
The Solution: The system takes that digital highlight and paints it directly onto the image the robot sees.
- If you need to grab the whole object, it draws a colored outline around it.
- If you need to touch a tiny, specific part (like the handle of a hammer), it paints a heat map (a glowing red spot) right on that handle.
Analogy: It's like a teacher pointing at a map and drawing a red circle around the city you want to visit, so the robot doesn't have to guess which city you meant.

3. The Inner Voice (Action-Level Conditioning)

The Problem: Sometimes, just showing the robot a picture isn't enough to make it move perfectly. It might still be a little confused.
The Solution: The system doesn't just show the robot the picture; it also whispers a secret instruction directly into the robot's "brain" (its action-generating code). It tells the robot, "Hey, ignore everything else, focus only on this specific glowing spot."
Analogy: It's like a coach not just pointing at the goal, but also tapping the player on the shoulder to say, "Run straight to that spot, not the one next to it."

What They Tested (The "Real-World" Proof)

The researchers tested this on a Unitree G1, which is a humanoid robot that looks like a person. They gave it 16 different tasks, including:

Picking the right cup when there are many similar ones (disambiguation).
Grabbing specific parts of an object, like the handle of a hammer instead of the head (fine-grained interaction).
Changing the target on the fly. Imagine the robot starts walking toward a red cup, but you suddenly look at a blue cup. The robot stops, realizes you changed your mind, and switches to the blue cup.

The Results

The paper claims that Gaze2Act was much better than robots that only listen to language or robots that try to guess based on text descriptions.

Language-only robots got confused easily (about 33% success rate on tricky tasks).
Gaze2Act got it right almost every time (about 89% success rate).
It was especially good at changing its mind when the user's gaze shifted, something the other robots struggled to do.

The Bottom Line

The paper concludes that gaze is a natural, low-effort way to tell a robot exactly what you want. It removes the guesswork. You don't need to be a robot programmer or speak in perfect code; you just need to look at what you want the robot to do, and the robot will follow your eyes.

Limitations mentioned in the paper:
The system isn't perfect yet. If you move your head too fast, or if something blocks your view (occlusion), the robot might get confused. Also, the system assumes you are looking at what you intend to touch, but sometimes people's eyes wander or look at things they don't actually want to grab.

Technical Summary: Gaze2Act

Problem Statement

While Vision-Language-Action (VLA) models have advanced robot learning through language instructions, they face a fundamental bottleneck in interactive manipulation: language alone is often insufficient to convey precise human intent. Users frequently struggle to explicitly describe which specific object to interact with among visually similar candidates, where exactly to act on an object (fine-grained interaction), or how to dynamically revise targets during execution. Existing approaches that translate language into spatial representations (e.g., bounding boxes or masks) inherit the intrinsic ambiguity of language and lack the temporal continuity required for real-time, dynamic intent updates. The core limitation identified is the absence of a direct, continuously updated, and spatially precise intent signal that complements static language instructions.

Methodology

The authors propose Gaze2Act, a framework that augments VLA policies with human gaze as a dynamic intent signal. The system operates on a Unitree G1 humanoid robot using a GROOT N1.5 backbone (featuring a frozen Eagle2 VLM and a Diffusion Transformer action head). The methodology proceeds through three integrated stages:

1. Cross-View Gaze Grounding

To bridge the gap between the human's first-person (ego) view and the robot's perspective, Gaze2Act employs a marker-free, training-free, and calibration-free semantic matching mechanism:

Feature Extraction: It utilizes DINOv3 (ViT-L/16) to extract multi-layer patch features from both the human's egocentric image and the robot's observation.
Coarse-Grained Matching: Upon a voice trigger, the system segments a reference object in the human view using SAM3. It then matches this reference against candidate masks generated in the robot view by averaging patch features and selecting the mask with the highest cosine similarity.
Fine-Grained Matching: To localize specific interaction points within the selected object, the system extracts a neighborhood feature vector around the gaze point in the human view and matches it against dense features within the coarse mask in the robot view.
Output: This process yields a unified spatial representation consisting of a target mask ( $m_t$ ) and a fixation point ( $p_t$ ).

2. Perception-Level Gaze Prompting

The grounded cues ( $m_t, p_t$ ) are rendered directly onto the robot's visual observation to provide explicit spatial guidance:

Contour Overlay: A colored contour is drawn along the boundary of the target mask to specify the object of interest, allowing for category-agnostic language instructions (e.g., "pick the outlined object").
Gaze Heatmap: The fixation point is converted into a 2D isotropic Gaussian heatmap centered at $p_t$ within the mask region. This provides a soft, robust signal for fine-grained part-level interactions, replacing brittle single-point inputs.

3. Action-Level Gaze Conditioning

To ensure the gaze signal directly influences the policy's decision-making beyond visual perception, the system injects gaze information into the action head:

Gaze Token Construction: The mask and fixation point are encoded into a compact spatial token ( $c^{gaze}_t$ ). This involves a position pathway (sinusoidal encoding) and an object pathway (using a frozen DINOv3 ViT encoder on the target crop).
Decoupled Cross-Attention: A new cross-attention branch is added in parallel to the existing vision-language path within the Diffusion Transformer (DiT). This branch injects the gaze token into the denoising process via separate Key/Value projections.
Stabilized Injection: The output projection of the gaze branch is zero-initialized. This ensures the added branch acts as a "no-op" initially, preserving the pretrained policy's prior knowledge while allowing the gaze influence to be learned gradually from the manipulation objective.

Key Contributions

Gaze as a Continuous Intent Interface: The paper proposes human gaze as a novel, continuous, and dynamic modality for VLA policies, enabling precise target specification across object-level, part-level, and dynamically evolving scenarios.
Gaze2Act Framework: A complete system that grounds first-person gaze into the robot's visual space without markers or calibration, integrating these cues via both perception-level visual prompting and action-level spatial conditioning.
Comprehensive Real-World Evaluation: Extensive testing on a Unitree G1 humanoid across seven task categories and 16 real-robot tasks, demonstrating consistent improvements over baselines in disambiguation, localization, and dynamic steering.

Experimental Results

The framework was evaluated on 16 tasks involving ambiguous objects, unseen objects, transparent objects, compositional tasks, subpart grasping, and dynamic intent steering.

Object Disambiguation: In tasks with ambiguous, unseen, or transparent objects, Gaze2Act achieved significantly higher intent accuracy (e.g., 92–100% on ambiguous objects) compared to language-only baselines (Vanilla GROOT: ~34%) and mask-based baselines (RoboGround/ControlVLA: ~60–80%).
Fine-Grained Interaction: For subpart grasping (e.g., hammer handle vs. head) and part-conditioned actions, Gaze2Act achieved an average intent accuracy of 80.4% and task success of 72.4%, whereas baselines could not provide part-specific conditions.
Dynamic Intent Steering: In long-horizon tasks where the target changes mid-execution, Gaze2Act successfully redirected the robot in 14 out of 30 trials, outperforming baselines (4–5/30) which struggled to update ongoing manipulation despite mask updates.
Ablation Studies: Results confirmed that both perception-level prompting and action-level conditioning are complementary. Visual prompting alone improved part-level success, while action-level conditioning improved object binding in compositional tasks. Zero-initialization of the gaze branch was shown to be critical for stable fine-tuning.

Significance and Claims

The paper claims that Gaze2Act demonstrates that human gaze provides a natural, low-burden, and highly expressive modality for human-in-the-loop VLA control. By leveraging the "eye-leads-hand" coordination inherent in human behavior, the framework overcomes the ambiguity and temporal discontinuity of language-only interfaces. The authors assert that their approach allows robots to attend to relevant regions and execute precise interactions under dynamic intent, effectively bridging the gap between high-level task description (language) and low-level spatial execution (gaze). The work highlights that gaze is not merely an observational signal but a direct manifestation of motor intention that can be seamlessly integrated into modern VLA architectures to enhance interactive manipulation capabilities.