🤖 AI

WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation

This paper introduces WatchAct, a comprehensive benchmark comprising 3,000 long-horizon instances that evaluates robot manipulation systems' ability to reason about observed human behavior through videos, revealing that current state-of-the-art models significantly struggle with tasks requiring event grounding, procedural reasoning, intent inference, and episodic tracking.

Original authors: Baiqi Li, Ce Zhang, Yu Fang, Yue Yang, Shangzhe Li, Mingyu Ding, Gedas Bertasius

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baiqi Li, Ce Zhang, Yu Fang, Yue Yang, Shangzhe Li, Mingyu Ding, Gedas Bertasius

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a kitchen and see a mess: a person opened a drawer, took out three spice jars to cook, and left them scattered on the counter. You tell a robot, "Put everything back where it came from."

For a human, this is easy. You remember what happened (the jars came from the drawer), in what order (they were taken out one by one), and why (to cook). You use the video of the person's actions in your mind to figure out the solution.

For today's robots, this is a nightmare. Most robot benchmarks only show the robot a single, static photo of the messy counter and ask it to clean up. They don't give the robot the "movie" of how the mess was made. Without that context, the robot is guessing.

WatchAct is a new test (a benchmark) designed to see if robots can actually watch a video of a human, understand what they did, and then clean up or continue the task based on that memory.

Here is a breakdown of how it works, using simple analogies:

1. The Test: "The Detective and the Actor"

Think of the robot as a detective and the video as a crime scene tape.

The Input: The robot gets a video of a human doing something (like moving a bowl) and a text instruction (like "Put the bowl back").
The Goal: The robot must figure out the hidden story. Did the human move the bowl because they were hungry? Did they move it to the left or right? Where did it originally come from?
The Scale: The test includes 3,000 different scenarios (like a massive library of short movies) covering 14 different types of challenges.

2. The Four "Brain Skills" Tested

The researchers organized the tasks into four categories, which are like different levels of detective work:

Event Grounding (Spotting the Clue): Can the robot watch a video and say, "Ah, at 3 seconds, the human picked up the red cup"? It's like finding the specific moment in a movie that matters.
Procedural Reasoning (Understanding the Recipe): Can the robot understand the order of events? If a human put a book on a shelf, then a cup on the table, the robot needs to know that sequence to reverse it or continue it.
Implicit Intent Inference (Reading the Mind): Sometimes humans don't say what they want; they just gesture. If a human points vaguely at a jar, can the robot guess, "Oh, they want me to open that"?
Episodic Reasoning (The Memory Test): Can the robot remember how the scene changed? If a human moved a chair from the corner to the middle, the robot needs to know the chair's "original home" to put it back.

3. How They Tested the Robots

The researchers didn't just ask the robot to "do it." They broke the robot's brain into two parts to see where it fails:

The Planner (The Brain): Can the robot watch the video and write a step-by-step plan? (e.g., "1. Pick up jar. 2. Move to shelf.")
The Doer (The Hands): If you give the robot a perfect plan, can it actually move its arms to do it?

They tested this in a computer simulation and on a real robot arm (a Franka Research 3).

4. The Results: "The Robots Are Still Learning"

The results were humbling. Even the smartest AI models available today struggled mightily.

The "Brain" Failed: When asked to watch a video and write a plan, the best AI (Gemini-3.1-Pro) only got 36.8% right. Humans got 97.1% right. The AI is essentially guessing on more than half the tasks.
The "Hands" Failed: Even when the researchers gave the robot a perfect plan (so the "brain" part wasn't the problem), the robot's "hands" (the policy) only succeeded 21.5% of the time. It's like having a perfect recipe but burning the cake because you don't know how to use the oven.
The Real World is Harder: When they tried this on a real robot arm, the success rate dropped even further to 14.0%.

5. Why Do They Fail?

The paper found three main reasons the robots are failing:

Long Stories: If the video is long and has many steps (like a 6-step cooking process), the robot gets confused and forgets the beginning by the time it reaches the end.
Perspective Issues: If the video is filmed from a weird angle (like from behind the human) or the instructions say "to the human's left," the robot gets lost. It can't translate "human left" to "robot left."
New Stuff: If the robot sees an object it hasn't seen before (like a new type of box), it freezes. It can't generalize its knowledge.

The Bottom Line

WatchAct is a reality check. It shows that while robots are getting good at moving things around in a static room, they are terrible at watching people, understanding the story behind the mess, and figuring out what to do next.

The paper concludes that we are far from having robots that can truly work alongside us in a home, because they still can't "watch and learn" the way humans do. They need to get much better at connecting the dots in a video before they can be trusted to help us in the kitchen.

Technical Summary: WatchAct – A Benchmark for Behavior-Grounded Robot Manipulation

Problem Statement

Robots operating in shared human environments must reason not only about the current state of a scene but also about the history of human actions, the sequence of events, and the unstated intents behind them. While language provides instructions, it often leaves critical spatial layouts, object histories, and procedural cues underspecified. Video, conversely, naturally captures these elements. However, existing manipulation benchmarks typically pair a language instruction with a single static image or treat video merely as a demonstration to be imitated. There is a lack of evaluation frameworks that require a robot to reason over observed human behavior to infer task-relevant events, procedural structures, and episodic changes before executing a manipulation task.

Methodology

The WatchAct Benchmark

WatchAct is a benchmark comprising 3,000 long-horizon instances across 14 distinct tasks organized into four capability domains derived from the cognitive demands of observing another agent:

Event Grounding: Identifying task-relevant actions, states, or moments (e.g., Fine-Grained Action, Count, Ordinal, State Change, Moment).
Procedural Reasoning: Recovering the temporal structure of action sequences (e.g., Imitation, Reversal, Temporal Sort).
Implicit Intent Inference: Inferring unstated goals from nonverbal cues or ambiguous references (e.g., Nonverbal Cue, Reference Disambiguation).
Episodic Reasoning: Tracking scene changes to support higher-level planning (e.g., Restore Previous State, Task Continuation, Error Correction, Conditional Execution).

Construction Pipeline:
Each instance pairs a real-world human-action video with a language instruction and an aligned, executable simulator task in LIBERO. The construction process involves:

Schema Generation: Using a frontier LLM (GPT-5.1) to generate filming guides and corresponding BDDL task definitions.
Data Construction: Recording human demonstrators following the guides and instantiating matching LIBERO tasks.
Alignment: Ensuring consistency between the video and simulator in terms of execution state, scene (object categories, counts, layouts via shared named regions), and goals.
Verification: A four-step quality check (Task Validity, Simulator Executability, Oracle-Plan Correctness, and Video-Simulator Alignment) ensures only valid instances are retained.

Spatial Variation:
To test spatial generalization, the benchmark varies camera viewpoints (frontal, side, oblique) and language reference frames (human-centered vs. camera-centered).

Evaluation Protocol

WatchAct introduces a disentangled evaluation protocol to isolate failure modes in the perception-to-action chain:

Video-to-Plan Reasoning: Measures the ability of Vision-Language Models (VLMs) to generate a high-level plan from the video and instruction. Success is measured by a symbolic executor checking if the plan satisfies the task's success conditions (Plan SR).
Policy Execution (Oracle Plans): Measures the ability of low-level policies to execute a ground-truth high-level plan without planning errors (Task SR).
Integrated Pipeline: Measures end-to-end performance where a VLM generates a plan and a policy executes it.

Key Contributions

Benchmark: WatchAct is the first benchmark to evaluate robot manipulation grounded in observed human behavior, featuring 3,000 instances across 14 tasks in four cognitively-motivated domains.
Disentangled Evaluation: A protocol that separately measures VLM reasoning, policy execution under oracle plans, and integrated pipeline performance, explicitly exposing where errors originate.
Empirical Findings: An evaluation of 9 VLMs and 4 policies in simulation and on a real Franka Research 3 robot, identifying specific bottlenecks in current systems.

Results

Performance in Simulation

Video-to-Plan Reasoning: Current VLMs struggle significantly. The best model, Gemini-3.1-Pro, achieves only 36.8% Plan SR, a gap of 60.3 points compared to human annotators (97.1%). Open-source models perform even lower (e.g., Qwen3-VL-235B-Thinking at 18.4%).
Policy Execution: Even with oracle plans, policies fail to reliably execute long-horizon tasks. The best policy, $\pi0.5$ , achieves only 21.5% Task SR. Other policies (OpenVLA, UniVLA, LingBot-VA) fall below 2%.
Integrated Pipeline: Errors compound in the full pipeline. The best integrated system (Gemini-3.1-Pro + $\pi0.5$ ) reaches only 16.3% Success Rate (SR).

Diagnostic Analysis

Long-Horizon Scaling: Performance degrades sharply as plan length increases. For GPT-5.4, Plan SR drops from 36.7% (2 actions) to 7.2% (6 actions).
Spatial Reasoning: Human-centered reference frames significantly degrade performance (e.g., a 19.5-point drop for GPT-5.4) compared to camera-centered frames.
Generalization: Policies fine-tuned on LIBERO data generalize poorly to out-of-domain scenarios (new objects/instructions). $\pi0.5$ drops from 47.1% Task SR (in-domain) to 10.6% (out-of-domain).

Real-Robot Experiments

Experiments on a Franka Research 3 robot with a Robotiq-2F85 gripper confirmed the simulation trends. The integrated pipeline achieved only 14.0% SR. The generalization gap persisted, with $\pi0.5$ dropping from 44.0% (in-domain) to 8.0% (out-of-domain) under oracle plans.

Significance and Claims

The authors claim that WatchAct serves as a critical testbed for the reasoning and control capabilities required for robots working alongside people. The paper asserts that:

Current systems are "far from solving" behavior-grounded manipulation.
The primary sources of failure are human-centered spatial reasoning, long-horizon planning, and physical execution reliability.
The disentangled protocol reveals that neither the planning stage (VLMs) nor the execution stage (policies) is currently robust enough for this task class.

The paper concludes by noting limitations, including the abstraction of contact dynamics in simulation (though real-robot trends matched), the structured nature of the human videos, and the restricted high-level action space, suggesting these as directions for future work.