💻 computer science

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

TimeLens2 introduces a generalist video temporal grounding framework that leverages multimodal LLMs with a novel multi-span supervision strategy and a temporal Wasserstein reward to achieve state-of-the-art performance across diverse benchmarks, significantly outperforming both size-matched baselines and much larger open-source models.

Original authors: Yuhan Zhu, Changlian Ma, Xiangyu Zeng, Xinhao Li, Zhiqiu Zhang, Songze Li, Jun Zhang, Tianxiang Jiang, Yuandong Yang, Ziang Yan, Zikang Wang, Xinyu Chen, Haoran Chen, Shaowei Zhang, Limin Wang

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhan Zhu, Changlian Ma, Xiangyu Zeng, Xinhao Li, Zhiqiu Zhang, Songze Li, Jun Zhang, Tianxiang Jiang, Yuandong Yang, Ziang Yan, Zikang Wang, Xinyu Chen, Haoran Chen, Shaowei Zhang, Limin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can watch any video in the world and describe exactly what's happening. It's like having a personal narrator who never misses a detail. But here's the catch: if you ask, "When does the dog catch the frisbee?" the robot might say, "The dog catches the frisbee!" and sound very confident, but it won't tell you when to pause the video to see it. It's like reading a book where the narrator describes the plot but refuses to give you the page numbers. Without those page numbers (or in this case, timestamps), you can't verify the story or find the evidence yourself. This is the problem of "temporal grounding." Scientists are trying to teach AI not just to describe videos, but to point to the exact seconds where the action happens, even if those seconds are scattered across a long movie or if the video is filmed from a shaky, first-person perspective.

Enter TimeLens2, a new AI model designed to be the ultimate video detective. While previous AI models were like detectives who could describe a crime scene but couldn't find the specific moment the suspect dropped the gun, TimeLens2 is trained to find the exact time intervals, whether it's a single second or a dozen scattered moments across a two-hour film. The researchers found that to make this work, they had to completely change how they taught the AI. Instead of just showing it a video and asking for an answer, they built a special "verification pipeline" where multiple AI agents act like a panel of judges, cross-checking each other's work to ensure the timestamps are real and not just guesses. They also invented a new way to grade the AI's answers that gives partial credit for being "close" even if the AI missed the exact start or end time, rather than just giving a zero for any mistake. The result? A compact AI model (as small as 2 billion parameters) that can find evidence in videos better than much larger, more expensive models, proving that with the right training, a small brain can be a sharper detective than a giant one.

The Detective's New Toolkit

So, how did TimeLens2 become so good at finding the "when" in video? The researchers realized that the old way of training AI was broken. Imagine trying to teach a student to find a needle in a haystack by just handing them the whole haystack and saying, "Find the needle." If the student guesses wrong, you just say "No," and they have no idea how close they were. In the world of video AI, this meant that if a model guessed the wrong second, it got zero credit, even if it was just one second off. This made learning very slow and frustrating.

To fix this, the team created TimeLens2-93K, a massive dataset built with a clever, multi-step process. Think of it like a production line for perfect video clues:

The Rough Draft: First, they used a smart AI to write a story (caption) for the whole video, breaking it down into smaller scenes.
The Question: From those scenes, they generated questions like, "When is the sauce being stirred?"
The Double-Check: Here's the magic part. Instead of trusting one answer, they sent the video to two different AI "detectives." These detectives worked independently to find the time intervals.
The Consensus: If both detectives agreed on the time, the answer was kept. If they disagreed, it was thrown out. This ensured that the training data was super reliable.
The Polish: Finally, they used a third, even smarter AI to sharpen the exact start and end times, making the boundaries precise.

This process turned a messy, unreliable dataset into a goldmine of 93,000 high-quality examples where the AI knows exactly when things happen.

The "Wasserstein" Score: A New Way to Grade

Once the AI had good data to learn from, the researchers had to teach it how to grade its own work. They introduced a new scoring method called the Temporal Wasserstein reward.

Imagine you are playing a game where you have to guess the location of a hidden treasure on a timeline.

The Old Way (tIoU): If you guess the wrong spot, you get a zero. It doesn't matter if you were one inch away or one mile away; you get nothing. This is like a teacher giving you an "F" for writing the wrong date, even if you were off by just one day.
The TimeLens2 Way (Wasserstein): This new method is like a GPS. If you are one inch away, it says, "You're almost there!" and gives you a high score. If you are a mile away, it says, "You're far off," and gives you a low score. It measures the distance between your guess and the truth.

This is crucial because it teaches the AI to keep trying to get closer, even when it hasn't found the exact answer yet. The paper shows that this method helps the AI recover from mistakes much faster, turning "silent" failures (where the AI gets a zero and learns nothing) into valuable learning moments.

The Results: Small Brain, Big Skills

The team tested TimeLens2 on seven different video challenges, ranging from short clips of people doing actions to long, complex videos filmed from a person's own eyes (like a GoPro on a helmet). They tested three versions of the model: a small 2-billion-parameter version, a medium 4-billion version, and a large 8-billion version.

The results were surprising. The tiny 2-billion version of TimeLens2 beat all other similarly sized AI models on every single test. Even more impressively, the 4-billion version outperformed massive, proprietary models that are hundreds of times larger (like a 397-billion parameter model). In simple terms, a small, well-trained TimeLens2 model is a better video detective than a giant, expensive one that hasn't had the same careful training.

For example, on a test called "MomentSeeker," where the AI has to answer questions like "What was put by the door?", the TimeLens2 models improved their scores significantly compared to their base versions. The 2B model jumped by 14.2 points, the 4B by 13.0 points, and the 8B by 18.1 points. This suggests that the secret sauce wasn't just making the AI bigger, but teaching it how to look for evidence more carefully and how to understand that "being close" is better than "being wrong."

Why This Matters

The paper concludes that by treating video evidence as a set of time intervals and using these new, smarter ways to train and grade the AI, we can make video search much more reliable. Instead of just getting a vague description, users can now get precise, verifiable timestamps. This turns video AI from a "black box" that guesses into a transparent tool that can show its work, making it possible to search through hours of footage to find exactly what you're looking for, whether it's a specific action, a recurring event, or a moment captured from a first-person view. The researchers suggest that this approach could make video archives searchable and trustworthy for everyone, from researchers to everyday users.

Technical Summary: TimeLens2

Problem Statement

Video Multimodal Large Language Models (MLLMs) have demonstrated the ability to describe video content, yet they frequently fail to identify when the supporting evidence occurs within a video. Without temporal grounding, video outputs remain unverifiable, forcing users to manually search timelines. The paper identifies "generalist video temporal grounding" as the core challenge: enabling a single model to predict a variable-cardinality set of evidence intervals across diverse video lengths, domains, query forms (declarative vs. interrogative), and viewpoints (third-person vs. egocentric).

The authors argue that existing approaches suffer from two structural mismatches:

Supervision Mismatch: Long-video labels often rely on brittle, one-pass global annotations that confuse similar occurrences, miss repeated evidence, or produce imprecise boundaries. Furthermore, many datasets focus on short videos and single-span answers, failing to address the complexity of multi-span grounding in long contexts.
Optimization Mismatch: Standard next-token prediction learns timestamp syntax but lacks an explicit interval-level objective. Reinforcement learning (RL) methods using temporal Intersection-over-Union (tIoU) as a reward fail to distinguish between "near misses" and distant errors (assigning zero reward to all disjoint predictions) and struggle with matching fragile one-to-one correspondences when predictions and targets have unequal cardinalities or fragmentation.

Methodology

TimeLens2 addresses these issues by treating temporal evidence as a first-class interval set throughout both supervision and optimization. The approach consists of two main components: a scalable data construction pipeline and a novel training framework.

1. TimeLens2-93K: Scalable Data Construction

To overcome the unreliability of long-video annotation, the authors propose a staged, evidence-verification pipeline that transforms brittle global annotations into reliable single- and multi-span supervision. The pipeline processes 34,867 diverse YouTube videos (stratified by duration from <1 min to >1 hour) through six steps:

Hierarchical Captioning: Videos are split into 20–60s clips using content-based boundaries. A large MLLM (Qwen3-VL) generates global captions for context and segment captions for specific actions, creating a time-stamped semantic index.
Query Synthesis: A query generator (Kimi-K2.5) uses these captions to create declarative queries and selects supporting segments, forming coarse proposals.
Dual-Agent Local Grounding: Two independent agents (Qwen3-VL-30B and TimeLens-8B) relocalize the proposed evidence directly from the video clips, returning interval sets or empty sets. This converts text matches into verifiable visual decisions.
Consensus & Verification:
- Temporal Consensus: Instances are retained only if the merged interval sets from both agents have a high IoU (>0.9), ensuring reproducibility.
- Semantic Verification: A text-video embedding similarity check (threshold >0.5) ensures the evidence is semantically relevant to the query.
Boundary Refinement: A final local refinement step observes a ±3s neighborhood around verified boundaries to sharpen transition points, merging gaps ≤1s to handle jitter.

This process yields TimeLens2-93K, a corpus of 93,232 grounding instances across 23,793 videos, including 12,091 multi-span examples.

2. Model Training: Capability Acquisition and Geometric Calibration

The training strategy separates learning to search from learning to calibrate boundaries:

Long-Context Supervised Fine-Tuning (SFT): The model (based on Qwen3-VL backbones) is fine-tuned on TimeLens2-93K, TimeLens-100K, and Ego4D-NLQ. The training emphasizes full-video context to force the model to isolate sparse evidence among distractors. To ensure robustness, the model is exposed to diverse instruction phrasings and response formats (JSON, natural language, etc.).
Reinforcement Learning with Temporal Wasserstein Reward:
- Rollout-Guided Mining: The model generates multiple off-policy completions; examples with low mean tIoU are up-weighted to focus optimization on unresolved errors.
- Reward Design: The authors introduce a composite reward $R = R_{tIoU} + R_{TW} - 1_{invalid}$ $R = R_{t I o U} + R_{T W} - 1_{in v a l i d}$ .
  - $R_{tIoU}$ : Measures the overlap of merged supports (target support already recovered).
  - $R_{TW}$ (Temporal Wasserstein): Computes the exact 1-Wasserstein distance ( $W_1$ ) between uniform distributions over the merged predicted and target interval supports. This provides dense, matching-free feedback for disjoint predictions and unequal cardinalities, ranking "near misses" higher than distant errors.
  - $1_{invalid}$ : Penalizes unparseable outputs.

Key Contributions

TimeLens2-93K: A staged evidence-verification pipeline that produces reliable single- and multi-span temporal grounding supervision for long videos, addressing the quality bottleneck in long-video annotation.
Temporal Wasserstein Reward: A matching-free reward based on exact $W_1$ distance over merged interval supports. It provides graded, fragmentation-invariant feedback for disjoint and unequal-cardinality predictions, solving the "zero-reward plateau" problem of tIoU.
Generalist Performance: The demonstration of compact 2B, 4B, and 8B models that achieve state-of-the-art performance across seven benchmarks covering long-video, multi-span, question-form, and egocentric grounding.

Experimental Results

TimeLens2 was evaluated on seven benchmarks: Charades-STA, ActivityNet Captions, QVHighlights, VUE-TR, VUE-TR-V2, MomentSeeker, and Ego4D-NLQ.

Performance:
- TimeLens2-2B outperforms all size-matched open-source baselines on all seven benchmarks, achieving an average mIoU of 44.5 (a 14.2 point improvement over its Qwen3-VL-2B backbone).
- TimeLens2-4B and TimeLens2-8B achieve state-of-the-art results, surpassing open-source models with up to 397B parameters (e.g., Qwen3.5-397B) and proprietary systems (e.g., Gemini 2.5 Pro) on multiple metrics.
- The 4B model improves over its backbone by 13.0 mIoU points on average, with significant gains in long-video (VUE-TR) and question-form (MomentSeeker) settings.
Ablation Insights:
- Data Quality: The progressive curation pipeline (consensus, verification, refinement) significantly boosts performance (from 42.0 to 45.8 mIoU) while reducing the dataset size by 7/8, proving that quality outweighs quantity.
- Reward Design: The Temporal Wasserstein reward ( $R_{TW}$ ) rescues 75.8% of "all-zero-tIoU" groups in RL, providing necessary learning signals where tIoU fails. It outperforms matched NGIoU (used in MUSEG) by 0.6 average mIoU, particularly in multi-interval scenarios.
- Long Context: Increasing the packed sequence length to 100K tokens yields the best results, specifically improving performance on long-video benchmarks like VUE-TR-V2.

Significance

The paper posits that for video MLLMs to become reliable interfaces to video archives, their outputs must be traceable to specific moments. TimeLens2 shifts temporal grounding from a specialized retrieval task to a native, auditable capability of generalist video MLLMs. By consistently treating evidence as an interval set from supervision through optimization, the framework enables compact models to outperform significantly larger systems in complex, real-world grounding scenarios. The work highlights that reliable long-video grounding requires not just scale, but a fundamental redesign of how evidence is verified and how geometric errors are penalized.