💬 NLP

TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs

This paper introduces TimeLens, a systematic framework for video temporal grounding that establishes a high-quality benchmark and training dataset (TimeLens-Bench and TimeLens-100K) while proposing efficient algorithmic designs like interleaved textual encoding and thinking-free RLVR to achieve state-of-the-art performance among open-source multimodal large language models.

Original authors: Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, Limin Wang

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, Limin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart friend who is amazing at describing what is happening in a movie. They can tell you, "A man is running," or "A dog is chasing a ball." But if you ask them, "When exactly does the man start running, and when does he stop?" they usually guess wildly. They might say, "He runs for the whole movie!" or "He runs for five seconds!" even if it was only three.

This is the problem TimeLens solves. It's a new way of teaching AI to understand time in videos, not just the pictures.

Here is the story of how they fixed it, explained simply:

1. The Broken Ruler (The Data Problem)

Imagine you are trying to teach a student how to measure a table. But the ruler you gave them is broken. Some marks are missing, some are in the wrong place, and some say "10 inches" when the table is actually 12 inches.

The researchers found that the "rulers" (datasets) everyone was using to test video AI were broken.

The Mess: The old tests had messy questions like "A man is running" (which happens 50 times in a video) or timestamps that were just wrong.
The Fix: The team went in with a fine-tooth comb. They manually re-watched thousands of videos and rewrote the questions and fixed the timestamps. They called this new, clean ruler TimeLens-Bench.
The Surprise: When they tested the old "smart" AI models on this new, honest ruler, many of them failed miserably! It turned out they were just cheating on the broken tests. The "proprietary" models (like GPT-5) actually did much better on the clean test, proving the old tests were lying.

2. The "No-Nonsense" Teacher (The Training Method)

Once they had clean data, they needed a better way to teach the AI.

The Old Way (Overthinking): Some researchers tried to teach AI to "think" before answering, like a student writing a long essay before solving a math problem. They thought, "If the AI thinks hard, it will get the time right."
The TimeLens Way (Instinct): The researchers discovered that for finding specific moments in a video, thinking is actually a distraction. It's like trying to catch a ball; if you stop to analyze the physics of the wind and your hand muscles, you'll miss it. You just need instinct.
The Result: They trained the AI to skip the "thinking" part and go straight to the answer. This made the AI faster, cheaper to train, and surprisingly, much more accurate.

3. The "Goldilocks" Recipe (Training Strategy)

They also figured out the perfect recipe for training, similar to cooking a steak:

Don't Overcook It (Early Stopping): In the past, people thought, "If a little training is good, a lot of training must be better!" They kept training the AI until it got tired and started making mistakes (overfitting). TimeLens found that you should stop training the moment the AI stops improving. It's like taking the steak off the grill the second it's perfect, not waiting until it's burnt.
The Right Difficulty (Sampling): They realized you shouldn't just feed the AI easy videos or impossible videos. You need to feed it videos that are just hard enough to challenge it, but not so hard that it gives up. This "Goldilocks" difficulty made the AI learn much faster.

4. The Result: A New Champion

By combining clean data (the fixed ruler) with instinctive training (no overthinking) and the perfect recipe, they built the TimeLens models.

The Achievement: These models are now the best open-source video time-keepers in the world.
The Shock: They are so good that they beat some of the most expensive, closed-source models from big tech companies (like GPT-5 and Gemini).

The Big Picture

Think of TimeLens as a master watchmaker. Before, everyone was trying to build a better watch using broken blueprints and confusing instructions. TimeLens said, "Let's fix the blueprints first, then teach the watchmaker to trust their gut, and stop when the watch is perfect."

Now, AI can finally answer the question: "When did that happen?" with the same confidence it answers "What happened?"

Based on the paper "TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs," here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

Video Temporal Grounding (VTG) is the task of localizing a specific event within a video based on a natural language query, outputting a start and end timestamp $(t_{start}, t_{end})$ . While Multimodal Large Language Models (MLLMs) excel at understanding what happens in a video, they often struggle with when it happens.

The paper identifies two critical bottlenecks hindering progress in VTG:

Data Quality Issues: Existing benchmarks (e.g., Charades-STA, ActivityNet Captions) and training datasets suffer from severe annotation errors, including inaccurate timestamps, duplicate queries, ambiguous descriptions, and events that do not exist in the video. These flaws lead to unreliable evaluation metrics that mislead research directions.
Suboptimal Algorithmic Design: There is a lack of systematic exploration into the best practices for optimizing MLLMs for VTG. Key questions remain unanswered regarding timestamp encoding strategies, the necessity of "thinking" (Chain-of-Thought) in reinforcement learning, and optimal training recipes.

2. Methodology

The authors propose TimeLens, a systematic framework addressing both data quality and algorithmic design.

A. Data Quality: Benchmark and Dataset Refinement

TimeLens-Bench (Evaluation): The authors manually audited three popular benchmarks (Charades-STA, ActivityNet Captions, QVHighlights). They established strict criteria for Query Clarity, Event Existence, Query Uniqueness, and Annotation Accuracy.
- They re-annotated thousands of samples, correcting errors like "multiple event occurrences" or "unclear queries."
- This resulted in a rigorously cross-validated evaluation suite.
TimeLens-100K (Training): Recognizing that training data is even noisier than evaluation data, they developed an automated re-annotation pipeline using advanced MLLMs (specifically Gemini-2.5-Pro) to generate high-quality queries and timestamps for a large-scale dataset of ~100k annotations.

B. Algorithmic Design Principles

The paper systematically explores three key dimensions:

Timestamp Encoding:
- Compared Position Embeddings (RoPE), Visual Overlays (OCR), and Textual Encoding.
- Finding: Interleaved Textual Encoding (inserting raw timestamp tokens like "10.2s" directly before visual frame tokens) outperforms complex alternatives while remaining simple and intuitive.
Training Paradigm:
- Compared Supervised Fine-Tuning (SFT), Thinking-based Reinforcement Learning with Verifiable Rewards (RLVR), and Thinking-free RLVR.
- Finding: VTG is primarily a perception-driven task, not a reasoning task. A Thinking-free RLVR approach (directly predicting timestamps without explicit reasoning steps) achieves superior performance and efficiency compared to SFT or thinking-based RLVR.
Training Recipes:
- Early Stopping: Training should stop when reward metrics plateau, as continued training leads to performance degradation.
- Difficulty-Based Sampling: Selecting training samples with sufficiently high difficulty (relative to the model's current capability) is crucial for maximizing performance gains.

3. Key Contributions

TimeLens-Bench: A high-quality, manually refined evaluation benchmark that reveals the unreliability of legacy benchmarks. It causes a dramatic re-ranking of models, showing that proprietary models often outperform open-source ones when evaluated correctly.
TimeLens-100K: A large-scale, high-quality training dataset created via automated re-annotation, addressing the noise in existing VTG corpora.
Algorithmic Insights:
- Identification of Interleaved Textual Encoding as the optimal timestamp representation.
- Demonstration that Thinking-free RLVR is the superior training paradigm for VTG, debunking the assumption that "thinking" is necessary for all complex tasks.
- Establishment of best practices for RLVR training (early stopping, difficulty-aware sampling).
TimeLens Models: A family of open-source MLLMs (TimeLens-7B and TimeLens-8B) that set new state-of-the-art (SOTA) records.

4. Results

Benchmark Performance: On the refined TimeLens-Bench, the TimeLens-8B model (based on Qwen3-VL-8B) achieves SOTA performance among open-source models.
- It significantly outperforms its baseline and other open-source competitors (e.g., Time-R1, VideoChat-R1).
- Crucially, it surpasses leading proprietary models such as GPT-5 and Gemini-2.5-Flash on the refined benchmarks.
Impact of Data Quality: Evaluation on original benchmarks showed open-source models artificially outperforming proprietary ones. On TimeLens-Bench, this trend reversed, with proprietary models showing their true strength, and TimeLens models narrowing the gap significantly.
Generalization: TimeLens-7B maintains strong general video understanding capabilities (tested on Video-MME) while excelling in temporal grounding, proving the method does not sacrifice general intelligence.

5. Significance

Correcting the Field: The paper exposes that previous progress in VTG was partly an illusion caused by noisy data. It provides a reliable foundation for future research.
Efficiency and Simplicity: By proving that "thinking" is unnecessary for VTG and that simple interleaved text encoding works best, the paper offers a more efficient and cost-effective path to building high-performance VTG models.
Open Science: The authors release all code, data (TimeLens-Bench, TimeLens-100K), and models, democratizing access to SOTA VTG capabilities and enabling further research into temporal awareness in MLLMs.

In summary, TimeLens redefines Video Temporal Grounding by prioritizing data integrity and identifying that perception-focused tasks benefit most from direct, thinking-free reinforcement learning, resulting in open-source models that rival or exceed the capabilities of top-tier proprietary systems.