💻 computer science

LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension

This paper introduces LongEgoRefer, a challenging new benchmark constructed from long-form Ego4D videos that addresses the limitations of existing short-clip datasets by featuring extreme target sparsity and complex interactions, revealing that current state-of-the-art models struggle significantly with long-form egocentric spatio-temporal grounding.

Original authors: Shunya Kato, Taiki Miyanishi, Shuhei Kurita, Mahiro Ukai, Nakamasa Inoue, Chenhui Chu

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shunya Kato, Taiki Miyanishi, Shuhei Kurita, Mahiro Ukai, Nakamasa Inoue, Chenhui Chu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a camera on your head that records your entire day, from the moment you wake up until you go to sleep. This results in a massive, unedited movie that is about 45 minutes long. Now, imagine someone hands you a specific sentence, like: "Find the moment where you pick up the blue mug with the chip on the handle and pour coffee into it."

Your job is to watch that 45-minute movie and pinpoint exactly when that happens and where in the frame the mug is.

This is the challenge the paper "LongEgoRefer" introduces. Here is a breakdown of what they did, using simple analogies:

1. The Problem: The "Needle in a Haystack"

Previous video games for AI were like looking for a needle in a small pile of hay. They used short video clips (a few seconds long) where the object was always visible, like a bright red ball bouncing in a room.

But real life isn't like that. In real life, the "needle" (the object you are looking for) might only appear for a few seconds in a 45-minute movie. It might be hidden behind your hand, or you might only touch it once before putting it away. The paper argues that current AI models are terrible at this because they are used to short, easy clips. They get lost in the long movie and forget what they are looking for.

2. The Solution: A New "Exam" for AI

The authors created a new benchmark called LongEgoRefer. Think of this as a new, much harder final exam for AI models.

The Source: They took thousands of hours of real-life "first-person" videos (from a dataset called Ego4D) where people are doing daily tasks.
The Questions: They wrote 1,498 specific questions (called "referring expressions"). These aren't simple like "Where is the cup?" They are complex stories like: "The clear glass jug, which was empty on the scale, was then filled with dark grains from a silver bag."
The Difficulty:
- Time: The videos are huge (average 45 minutes).
- Sparsity: The object they are asking about might only show up for 1% of the video. It's like asking someone to find a specific 10-second conversation in a 45-minute radio broadcast.
- Complexity: The descriptions require understanding how objects move and interact with hands, not just what they look like.

3. The Test: How Did the AI Do?

The authors tested the smartest AI models available (including giants like GPT-5 and Gemini) on this new exam.

The Result: The AI models struggled significantly. Even the most advanced models got very low scores.
The Analogy: It's like giving a student a math test where they have to remember a specific number from a book they read 40 minutes ago, while the book is constantly flipping pages. Most models got lost in the middle of the book.
The Winner: The model named GPT-5 performed the best, but even it only got about 16% of the "spatio-temporal" (time and place) details correct. This proves the task is incredibly hard, even for the best technology we have right now.

4. Why Did They Fail?

The paper found two main reasons the AI failed:

Memory Loss: The models couldn't keep track of the whole 45-minute story. They forgot what the object looked like by the time they reached the end of the video.
The "When" vs. "Where" Trap: The AI has to figure out when the event happens first, and then where the object is. If the AI guesses the wrong time (e.g., "It happened at minute 10" when it actually happened at minute 30), it fails completely, even if it knows what the object looks like. It's like trying to find a specific page in a book but starting your search on the wrong chapter.

5. The Takeaway

The paper concludes that we need a new generation of AI that can truly "watch" long videos without getting confused or forgetting details. They have released this new "exam" (the benchmark) and the code so other researchers can try to build better models to solve this "needle in a haystack" problem.

In short: They built a super-hard test using real-life, long videos to show that today's AI is still bad at finding specific, rare moments in long streams of video, and they are challenging the world to fix it.

Technical Summary: LongEgoRefer

Problem Definition

The paper addresses the limitations of existing Video Referring Expression Comprehension (Video REC) benchmarks, which primarily focus on short, trimmed video clips (typically 3–10 seconds) where target objects appear densely and predictably. These settings fail to reflect real-world egocentric (first-person) recordings, which are characterized by:

Long-form duration: Continuous recordings spanning tens of minutes to hours.
Sparse object occurrence: Target objects appear transiently, often for less than 2% of the total video duration.
Complex interactions: Dynamic human-object interactions (HOI) that require understanding temporal context and state changes.
Linguistic complexity: Referring expressions that are long, descriptive, and necessary to disambiguate targets within cluttered, visually similar scenes.

The core challenge is Spatio-Temporal Grounding in untrimmed, long-form egocentric videos: given a natural language query, a model must identify the precise temporal interval ( $t_{start}, t_{end}$ ) and the corresponding spatial bounding boxes ( $B$ ) for the referred object, despite the object being absent for the vast majority of the video.

Methodology

Benchmark Construction (LongEgoRefer)

The authors constructed LongEgoRefer using the Ego4D dataset and EgoTracks annotations. The construction pipeline involved:

Data Filtering: Selecting videos longer than 20 minutes and object tracks lasting at least 10 seconds where the camera wearer actively manipulates the object ( $is\_active = True$ ).
Hybrid Annotation: A semi-automated pipeline was used to generate referring expressions:
1. LLM Generation: Gemini 2.5 Flash generated 10 candidate descriptions per clip, which were then summarized.
2. Human Curation: Expert annotators rigorously verified and refined the expressions. Approximately 95.39% of the initial LLM-generated descriptions were substantially revised to ensure factual accuracy, temporal exclusivity (the description applies only to one specific moment), and uniqueness.
Dataset Statistics: The final benchmark contains 1,498 referring expressions across 433 videos. The average video duration is 45 minutes (2,714.9 seconds), and the average referring expression length is 63.5 words. The target object appears in only 1.3% of frames on average.

Baselines and Evaluation

To evaluate the difficulty of the benchmark, the authors established two types of baselines:

Adapted Video REC Models: Existing end-to-end models (e.g., VideoLISA, Sa2VA, Grounded SAM2) designed for short clips were adapted using a sliding window inference strategy. The long video is partitioned into short windows, and the model processes each independently.
Training-Free Two-Stage Pipeline: A modular approach decoupling the task into:
- Temporal Grounding: Using a Vision-Language Model (VLM) to identify the coarse temporal interval matching the query.
- Spatial Grounding: Using Grounded SAM2 (Grounding DINO + SAM2) to localize the object within the identified clip.

Metrics: The evaluation employs standard video REC metrics including temporal IoU (tIoU), video IoU (vIoU), spatio-temporal IoU (STIoU), and a modified IoU for negative frames (IoU+n).

Key Results

Extensive experiments were conducted on 4 video REC models, 14 open-source VLMs, and 5 closed-source VLMs.

Overall Difficulty: Even state-of-the-art models struggle significantly. The best-performing model, GPT-5, achieved a temporal IoU at 0.1 threshold (tIoU@0.1) of 37.31 and a mean spatio-temporal IoU (mSTIoU) of 8.76. Most open-source models scored in the single digits for tIoU@0.1.
Open vs. Closed Source: A substantial performance gap exists. Closed-source models (GPT-5, Gemini 2.5 Pro) significantly outperformed open-source counterparts (e.g., InternVL 3.5, Qwen3-VL), though all models showed modest absolute performance.
Error Propagation: The two-stage pipeline revealed that errors in temporal grounding (Stage 1) strictly constrain spatial grounding (Stage 2). When provided with oracle (ground-truth) temporal segments, spatial grounding performance (mvIoU) improved dramatically (e.g., from 7.89 to 43.54 for GPT-5), highlighting that temporal localization is the primary bottleneck.
Impact of Variables:
- Temporal Resolution: Reducing frame sampling (e.g., to 64 frames) caused a sharp performance drop, indicating the need for dense visual cues.
- Object Sparsity: Performance degraded drastically as the object's appearance rate dropped below 1%.
- Video Length: Performance generally decreased as video duration increased, though models with massive context windows (Gemini 2.5 Pro) showed more robustness.
- Linguistic Complexity: Longer, more complex referring expressions correlated with lower performance.

Significance and Claims

The paper claims that LongEgoRefer establishes a new, more realistic standard for evaluating video understanding by bridging the gap between current benchmarks and real-world egocentric scenarios.

Intrinsic Difficulty: The benchmark demonstrates that accurately localizing rare events and objects in long-form videos remains a significant, unresolved challenge for current state-of-the-art models, even those with advanced reasoning capabilities.
Limitations of Current Paradigms: The results expose fundamental bottlenecks in existing approaches:
- Sliding-window strategies lack global temporal awareness, leading to high false positives.
- Two-stage pipelines suffer from critical error propagation where temporal inaccuracies prevent successful spatial localization.
Future Direction: The authors emphasize the need for models with robust long-term memory, advanced interaction reasoning, and the ability to ground complex language in sparse, cluttered visual environments. They note that while LongEgoRefer serves as a rigorous testbed, developing large-scale training datasets for such long-form videos remains a necessary future step.

The paper concludes that LongEgoRefer is not merely a dataset but a "demanding spatio-temporal grounding problem" designed to drive research toward more capable, human-centric AI systems.