🤖 AI

ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos

This paper introduces ToG-Bench, the first task-oriented spatio-temporal video grounding benchmark for egocentric videos, which addresses the limitations of existing object-centric approaches by featuring task-oriented, explicit-implicit dual, and one-to-many grounding scenarios to better evaluate and advance embodied intelligence.

Original authors: Qi'ao Xu, Tianwen Qian, Yuqian Fu, Kailing Li, Yang Jiao, Jiacheng Zhang, Xiaoling Wang, Liang He

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qi'ao Xu, Tianwen Qian, Yuqian Fu, Kailing Li, Yang Jiao, Jiacheng Zhang, Xiaoling Wang, Liang He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to help you around your house. You don't just want the robot to say, "I see a red cup on the table." You want it to understand your intent. If you say, "I'm thirsty, get me some water," the robot needs to figure out that it must find the sink, turn on the faucet, and grab a glass—even though you never explicitly mentioned the faucet or the glass.

This is exactly what the paper ToG-Bench is about. It's a new "exam" designed to test how well AI robots can understand human tasks from a first-person perspective (like wearing a camera on your head).

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Descriptive" Robot vs. The "Task" Robot

Previous AI tests were like playing a game of "I Spy."

The Old Way: You say, "Find the blue pillow." The robot looks for something blue and fluffy. It's easy because the robot just matches words to pictures.
The Real World: You say, "Make coffee." The robot needs to know that "making coffee" involves a machine, beans, a filter, and a mug. It has to figure out what to touch and when to touch it, even if you didn't list every single item.

The authors realized that current AI is great at "I Spy" but terrible at "Make coffee." It doesn't understand the goal, only the description.

2. The Solution: ToG-Bench (The New Exam)

The team created a new benchmark called ToG-Bench (Task-Oriented Grounding). Think of this as a driving test for robots. Instead of just asking them to "spot a stop sign," they ask them to "navigate to the grocery store and buy milk."

The exam has three tricky rules:

The "Hidden Clue" Rule (Explicit vs. Implicit): Sometimes you say "Turn on the light" (Explicit: the light switch). Sometimes you say "It's dark in here" (Implicit: the robot must realize it needs to find the light switch, even though you didn't say "switch"). The robot must use common sense to fill in the blanks.
The "Teamwork" Rule (One-to-Many): A single instruction often needs multiple tools. If you say "Clean the whiteboard," the robot needs to find the board, the marker, and the eraser. It can't just find one; it needs the whole crew.
The "Time Travel" Rule (Spatio-Temporal): The robot doesn't just need to find the object; it needs to know when to interact with it. It has to track the object as you move around the room, not just in a single snapshot.

3. How They Built It (The Assembly Line)

Building this exam was hard because you can't just ask a human to write 2,700 complex instructions for 100 videos. That would take forever.

The Pipeline: They used a "smart assistant" (a powerful AI called Gemini) to watch videos and suggest tasks like "Wash your hands."
The Safety Check: Then, real humans stepped in as editors. They checked: "Is the faucet actually visible? Is the instruction safe? Did the robot track the water correctly?"
The Result: A high-quality dataset of 100 videos with over 2,700 tasks, covering everything from simple actions to complex, multi-step chores.

4. The Results: The "Smart but Clumsy" Robot

The authors tested the world's best AI models (like GPT-5 and Gemini) on this new exam. The results were a mix of good news and bad news:

The Good News: The AIs are very smart at understanding the idea. If you say "Make coffee," they can usually tell you, "Okay, I need a coffee machine." They are great at the "What" and the "Why."
The Bad News: They are terrible at the "Where" and "When."
- The Drift: Imagine a robot trying to hold a cup while walking. The AI knows it's holding a cup, but as the camera moves, the AI loses track. It might point to the wall instead of the cup.
- The "Hidden" Struggle: When the task required the robot to guess an object you didn't mention (like the faucet), the AI's performance dropped significantly.
- The Complexity Crash: When asked to find three things at once (like a computer, mouse, and keyboard), the AI got confused and failed to find them all correctly.

The Big Takeaway

This paper is a wake-up call. We have built AIs that can read a recipe and understand the story, but we haven't taught them how to actually cook in a real kitchen while moving around.

ToG-Bench is the first step in fixing this. It shows us exactly where the robots are failing so engineers can build the next generation of AI that doesn't just "see" the world, but truly understands how to interact with it to get things done.

1. Problem Statement

Current Spatio-Temporal Video Grounding (STVG) research primarily focuses on object-centric and descriptive instructions (e.g., "localize the red cup"). While effective for static or exocentric (third-person) views, these benchmarks fail to address the needs of embodied agents operating in egocentric (first-person) environments.

The core limitations identified are:

Lack of Task Orientation: Existing benchmarks do not require agents to infer objects based on high-level goals (e.g., "make coffee" requires identifying a coffee machine, not just describing a visible object).
Absence of Implicit Reasoning: Current tasks rarely require inferring objects that are not explicitly mentioned but are functionally necessary (e.g., "turn on the sink" implies the existence of a faucet).
Single-Object Focus: Most benchmarks assume a one-to-one mapping between instruction and object, ignoring the multi-object coordination required for complex tasks (e.g., "prepare the whiteboard" requires a board, marker, and eraser).
Egocentric Dynamics: Existing datasets often lack the specific challenges of first-person views, such as rapid camera motion, self-occlusion, and attention shifts.

2. Methodology

A. Task Definition: Task-oriented Spatio-Temporal Video Grounding (T-STVG)

The authors formally define T-STVG as a mapping function $F: (G, V) \rightarrow \{(O_i, T_i)\}_{i=1}^n$ , where:

$G$ : A high-level natural language goal.
$V$ : An egocentric video sequence.
Output: A set of $n$ task-relevant objects $\{O_i\}$ and their dense spatio-temporal trajectories $T_i$ (bounding boxes over time).

The task is characterized by three distinct challenges:

Task-Driven Reasoning: Inferring objects based on functional roles rather than visual descriptions.
Explicit–Implicit Dual Grounding: Handling objects explicitly mentioned in the prompt and those implicitly inferred via context/commonsense.
One-to-Many Grounding: Localizing multiple interrelated objects simultaneously for a single instruction.

B. Dataset Construction: ToG-Bench

ToG-Bench is the first benchmark designed for this task, constructed via a semi-automated pipeline combining foundation models and human refinement.

Source: 100 video clips (avg. 87.88s) from ScanNet (indoor, first-person trajectories).
Scale: 2,704 task-oriented instructions and 4,194 annotated object instances.
Annotation Pipeline (Top-Down Approach):
1. Instruction Generation: Uses Gemini 2.5 Pro to generate task-oriented instructions and object descriptions (both explicit and implicit) based on video captions.
2. Grounding & Tracking: Uses Grounding-DINO for open-vocabulary detection and SAM2 for temporal tracking to generate spatio-temporal bounding box tubes.
3. Human Verification: Rigorous manual filtering to remove hallucinations, tracking drift, and occlusion issues.
Statistics: The dataset features a near-balanced split between explicit (51.26%) and implicit (48.74%) tasks. Approximately 50% of tasks involve multiple objects (1-to-many).

C. Evaluation Metrics

The authors propose a decoupled evaluation paradigm to separate reasoning from localization:

Reasoning Capability (Recognition): Measured by Accuracy (Acc) in identifying the correct object category (Cosine similarity of text embeddings).
Grounding Capability (Localization):
- Temporal: Recall (R1@ $\tau$ ) and Mean Temporal IoU (m_tIoU).
- Spatial: Average Precision (AP@ $\tau$ ) and Mean Video IoU (m_vIoU).
Granularity: Metrics are calculated at both Object-level (per entity) and Task-level (strict "all-or-nothing" success for the entire instruction).

3. Key Contributions

ToG-Bench Benchmark: The first dataset for task-oriented STVG in egocentric videos, introducing the concepts of implicit reasoning and multi-object grounding.
Task-Level Metrics: A novel evaluation suite that jointly assesses semantic recognition and spatio-temporal precision, distinguishing between explicit and implicit task success.
Comprehensive Benchmarking: Systematic evaluation of 7 state-of-the-art MLLMs (including GPT-5, Gemini 2.5 Pro, Qwen3-VL, InternVL) and specialized grounding baselines (Grounded-SAM2).

4. Experimental Results

The authors evaluated models in a zero-shot setting. Key findings include:

Performance Hierarchy: There is a significant gap between Task Accuracy (identifying what to do) and Grounding Precision (identifying where/when).
- Example: GPT-5 achieves 89.42% Task Accuracy but only 10.22% Spatial Video IoU.
- Example: Gemini 2.5 Pro shows superior spatial-temporal alignment (15.42% m_vIoU) compared to GPT-5, despite lower reasoning accuracy.
The Implicit Reasoning Gap: All models struggle significantly with implicit tasks.
- GPT-5 drops from 95.60% accuracy on explicit tasks to 86.98% on implicit tasks.
- Open-source models (e.g., Qwen3-VL-8B) show a massive drop in spatial grounding for implicit tasks (down to ~1.15% m_vIoU).
Multi-Object Complexity: Performance degrades sharply as the number of required objects increases.
- For tasks with 3+ objects, GPT-5's task accuracy drops from ~96% (1 object) to 75%, and spatial precision collapses.
- Open-source models often fail completely on multi-object tasks (e.g., Qwen3-VL-8B drops to 29% task accuracy).
Video Duration Impact: While semantic reasoning remains stable over long videos, spatial and temporal grounding precision degrades significantly as video length increases due to tracking drift and accumulated perceptual errors.
Specialist vs. Generalist: Specialized grounding models (Grounded-SAM2) excel at spatial precision when given object names but fail at temporal reasoning and task inference. Conversely, MLLMs excel at reasoning but lack fine-grained spatial alignment.

5. Significance and Future Directions

Bridging Perception and Interaction: ToG-Bench highlights that current MLLMs are strong at "understanding" but weak at "acting" (precise localization) in dynamic, embodied scenarios.
Limitations of Current Architectures: The results suggest that standard MLLMs lack coherent mechanisms to jointly reason about functional roles and dynamic visual context. They struggle with the "what-when-where" triplet under rapid viewpoint shifts.
Future Work: The benchmark provides a rigorous testbed for developing next-generation embodied AI agents capable of handling implicit goals, multi-object coordination, and long-horizon tasks. The authors plan to release the data and code to facilitate further research.