🤖 AI

TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

This paper introduces TAVIS, a comprehensive benchmark and evaluation infrastructure for active-vision imitation learning featuring diverse task suites, embodied platforms, and novel metrics like Gaze-Action Lead Time to systematically quantify the benefits and limitations of anticipatory gaze in robotic manipulation.

Original authors: Giacomo Spigler

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giacomo Spigler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Giving Robots "Eyes" That Move

Imagine you are trying to teach a robot to build a sandwich. Most robots today have a camera mounted on a wall or a stand, like a security camera. They can see the whole table, but they can't look closer at the mustard bottle or peek under a napkin. They are stuck with a "fixed gaze."

This paper introduces TAVIS, a new way to test robots that can actually move their eyes (or head and wrists) to look where they need to. The authors created a "gym" (a benchmark) to see if letting a robot control its own gaze actually helps it learn better than just staring at a fixed spot.

The Two "Gyms": TAVIS-Head and TAVIS-Hands

The researchers built two different training environments to test two different ways robots can move their eyes:

TAVIS-Head (The "Neck" Gym):
- The Scenario: Imagine a messy table with a red cube hidden among ten other toys. Or a card that says "Pick the left one" or "Pick the right one."
- The Challenge: The robot has to turn its head (like a human looking around a room) to find the right object or read the clue.
- The Test: They compare a robot that can turn its head against a robot that is forced to stare straight ahead.
- The Result: Moving the head helps a lot when the robot needs to search for something or read a clue. But if the table is already perfectly visible, moving the head doesn't help much and can sometimes even be a distraction.
TAVIS-Hands (The "Wrist" Gym):
- The Scenario: Imagine a box with a lid that is closed, or a screen blocking the view of an object. The robot's main "head camera" can't see what's inside or behind the screen.
- The Challenge: The robot has to use cameras mounted on its wrists to "peek" around corners or inside boxes.
- The Test: Can the robot use its wrist cameras to see what its head camera cannot?
- The Result: Yes. When the view is blocked, the robot must use its wrist cameras to succeed.

The "Crystal Ball" Metric: GALT

One of the coolest parts of this paper is a new way to measure how the robot looks, not just if it succeeds.

The Human Habit: Think about when you reach for a cup of coffee. Your eyes usually lock onto the cup before your hand even starts moving. You look first, then reach. This is called "anticipatory gaze."
The New Metric (GALT): The authors created a score called Gaze-Action Lead Time (GALT). It measures the time difference between when the robot's eyes land on the target and when its hand grabs it.
The Finding: When they trained robots just by watching humans (imitation learning), the robots naturally learned to look at the object before grabbing it. Their "look-ahead" time was almost exactly the same as the human who taught them. The robot learned to be "legible"—meaning a human watching the robot can tell what it's about to do before it actually does it.

The "Stress Test": What Happens When Things Change?

The researchers didn't just test the robots in the exact same room every time. They played tricks on them:

Moving the objects: They moved the toys to spots the robot had never seen before.
Moving the robot: They moved the robot's starting position slightly.

The Result: The robots got much worse at their jobs when the environment changed. Even though they were smart enough to move their eyes, they struggled to adapt when the world looked different than what they practiced on. This shows that while active vision is helpful, it doesn't make robots "superhuman" at handling surprises yet.

The Takeaway

This paper isn't about building a robot that can do your laundry tomorrow. It's about building a fair scoreboard for scientists.

Before TAVIS, every robot lab had its own rules, its own robot, and its own camera setup. It was impossible to say, "Robot A is better than Robot B." TAVIS provides a standardized set of tasks and a standardized way to measure success (including that "look-before-you-leap" timing).

In short:

Active vision (moving eyes) helps, but only when the robot actually needs to look around or peek at hidden things.
Robots learn to look first just like humans do, simply by copying us.
Robots are still fragile: If you move the furniture, they get confused.

The authors have released all their code, data, and trained robots to the public so other scientists can use this "gym" to train and test their own robots, hopefully leading to machines that are better at seeing and understanding the world.

Technical Summary: TAVIS – A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

Problem Statement

Imitation learning (IL) for visuomotor manipulation has advanced rapidly, yet most existing benchmarks and methods rely on fixed third-person or wide-angle cameras. While recent independent systems have demonstrated that policies controlling their own gaze (active vision) improve manipulation performance, the field lacks a shared evaluation infrastructure. Current benchmarks (e.g., LIBERO, RLBench, CALVIN) assume fixed cameras and cannot isolate active vision as a controlled variable. Furthermore, existing active-vision artifacts are often hardware-locked to specific real-robot setups, preventing cross-system comparison. Consequently, the community cannot systematically quantify what active vision contributes, on which task types, or under what conditions. Additionally, no current benchmark measures whether learned policies acquire anticipatory gaze—the human-like behavior where gaze precedes hand movement to signal intent—a critical dimension for legibility in Human-Robot Interaction (HRI).

Methodology

The authors introduce TAVIS (Tilburg Active Vision Imitation Learning), a simulation-based evaluation infrastructure built on IsaacLab and IsaacLab-Arena. TAVIS is designed to isolate the roles of active vision through two complementary task suites and two humanoid torso embodiments (GR1T2 and Reachy2).

1. Task Suites

TAVIS-Head (Global Search): Contains five tasks requiring head reorientation (pan/tilt necks) for global search, clutter disambiguation, and intent signaling. Tasks include conditional-pick (looking at a cue card), wait-then-act (temporal monitoring), clutter-pick-cube, clutter-pick-lift, and multi-shelf-scan.
TAVIS-Hands (Local Occlusion): Contains three tasks where head movement is insufficient due to occlusion, requiring wrist-mounted cameras. Tasks include peeking-box (determining an open side of a box), occluded-reach (reaching behind a screen), and blocked-clutter-pick-cube (locating an object with the head camera masked).

2. Data Collection and Protocol

Demonstrations: Approximately 2,200 episodes were collected via first-person teleoperation using a Meta Quest 3. The operator controls the robot's arms and neck (gaze) simultaneously.
Paired Recording: Crucially, every episode is recorded with both a head-mounted camera (active) and a fixed workspace camera (passive) simultaneously. This allows for a paired comparison where the only variable is the camera mode, holding all other factors (demonstration, scene layout, trajectory) constant.
Distribution Splits: The benchmark includes In-Distribution (ID) splits and Out-of-Distribution (OOD) splits (ood-spatial for object placement, ood-init-pose for robot reset pose) to test generalization and robustness against memorization.

3. Evaluation Primitives

TAVIS introduces three specific evaluation axes:

Headcam vs. Fixedcam: A direct comparison of policy performance using active vs. passive vision on identical trajectories.
GALT (Gaze-Action Lead Time): A novel metric grounded in cognitive science and HRI. It quantifies the temporal gap between the head reaching a final fixation ( $t_{head}$ ) and the completion of a grasp ( $t_{hand}$ ). A positive GALT indicates anticipatory gaze. The metric is derived solely from proprioception (commanded actions), making it portable to real-robot evaluation.
ID/OOD Splits: Extending the LIBERO-Pro paradigm to active vision to distinguish interpolation from extrapolation.

Key Contributions

The TAVIS Benchmark: A composable platform featuring two task suites, two robot embodiments, and a unified 19-dimensional canonical action space. It releases ~2,200 episodes of paired headcam/fixedcam demonstrations in LeRobot v3.0 format.
New Evaluation Primitives:
- A paired headcam-vs-fixedcam protocol to isolate active vision benefits.
- GALT, a kinematic metric to quantify anticipatory gaze, bridging the gap between robot learning and cognitive science/HRI literature.
- Controlled ID/OOD distribution splits to stress-test generalization.
Baseline Analyses: Experiments using Diffusion Policy and $\pi_0$ (Vision-Language-Action models) provide the first systematic comparison of active-vision IL.

Experimental Results

The authors trained multi-task and single-task policies on TAVIS and reported the following findings:

Active Vision Benefits are Task-Conditional: Active vision generally improves performance, but the gains are not uniform. On TAVIS-Head, head-mounted cameras outperformed fixed cameras (e.g., +28 percentage points on conditional-pick for GR1T2). However, benefits were negligible or negative on tasks where the workspace was already fully observable (e.g., wait-then-act on GR1T2 showed a -23pp regression). On TAVIS-Hands, where head cameras are structurally uninformative, wrist cameras were essential, achieving 70–77% success rates in ID settings.
Multi-Task Training: Multi-task policies outperformed single-task baselines, particularly on the TAVIS-Hands suite, suggesting that active vision benefits from shared representations across tasks.
Sensitivity to Distribution Shift: Policies degraded sharply under controlled OOD shifts. For TAVIS-Head, success rates dropped from 43.0% (ID) to 6.6% under ood-init-pose perturbations. The degradation was more severe for head-mounted cameras than fixed cameras in certain OOD scenarios, highlighting the fragility of active-vision policies to pose shifts.
Acquisition of Anticipatory Gaze: Policies trained solely via imitation (without explicit GALT optimization) acquired anticipatory gaze. The median GALT for learned policies was comparable to the human teleoperator reference (approx. 2–3 seconds lead time), with deviations within ~7–10% for most tasks. This suggests that the temporal coupling of gaze and action is an emergent property of high-quality imitation.

Significance and Claims

The paper positions TAVIS as a necessary infrastructure to overcome the current bottleneck in active-vision research. By providing a shared benchmark with controlled variables, it enables fair comparison of disparate systems that previously could not be compared due to differing hardware and task definitions.

The authors claim that TAVIS demonstrates:

That active vision is beneficial but task-conditional, not a universal upgrade.
That current IL policies are fragile under distribution shifts, particularly when relying on active gaze.
That imitation alone is sufficient to learn anticipatory gaze patterns that mirror human behavior, validating the use of teleoperation data for training legible robotic agents.

The benchmark is designed to be extensible, allowing for the addition of new robots, tasks, and OOD axes (e.g., visual textures, scene-level changes) to further explore the limits of egocentric active vision.