💻 computer science

MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues

This paper reveals that Multimodal Large Language Models possess latent temporal grounding capabilities in their prefill attention but fail to utilize them during generation, prompting a training-free inference framework that extracts these attention cues to restrict visual context and significantly improve video temporal grounding performance.

Original authors: Dazhao Du, Liao Duan, Jian Liu, Tao Han, Yujia Zhang, Eric Liu, Xi Chen, Song Guo

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dazhao Du, Liao Duan, Jian Liu, Tao Han, Yujia Zhang, Eric Liu, Xi Chen, Song Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Know-It-All" Who Forgets

Imagine you have a very smart, well-traveled friend (a Multimodal Large Language Model, or MLLM) who can describe a video perfectly. You show them a video of a man fixing a car and ask, "When does he start tightening the bolt?"

Your friend looks at the whole video, thinks for a second, and says, "Oh, that happens between 2:00 and 2:30." But they are wrong. The bolt was actually tightened between 1:10 and 1:20.

The researchers in this paper discovered something surprising: Your friend actually did know the right time, but they forgot it by the time they spoke.

The Discovery: The "Internal GPS" vs. The "Noisy Room"

The team peered inside the model's "brain" (its attention mechanism) and found a split personality:

The "Prefill" Phase (Reading the Menu): When the model first reads the question and scans the video, a tiny, special group of neurons (called Temporal Grounding Heads) acts like a laser-focused GPS. It locks onto the exact seconds where the action happens. At this moment, the model is 100% sure of the answer.
The "Decoding" Phase (Ordering the Meal): As the model starts typing out its answer word-by-word, it gets distracted. It forgets the GPS signal and starts looking at the most visually loud parts of the video. If the video has a bright red car in the background, the model might get confused and say, "Oh, the action must be when the red car is visible!" even if the red car has nothing to do with the bolt.

The Analogy: Imagine you are in a crowded, noisy room (the video). You spot your friend (the event) clearly for a split second. But then, a loud band starts playing (distractors), and your friend gets lost in the crowd. By the time you try to tell someone where your friend is, you point to the band instead.

The Solution: "Read, Then Re-Speak"

The authors didn't try to retrain the model (which is expensive and slow). Instead, they built a clever "inference-time framework" that acts like a smart editor.

Here is how their two-step process works:

Step 1: The "Sniff Test" (Read)
Before the model is allowed to give its final answer, the system checks the "GPS signal" from those special neurons during the reading phase.

It asks: "Did the model actually find the right spot?"
If the model seems confident and the GPS signal matches the answer, it just lets the model speak.
If the model seems confused or the GPS signal is weak, the system says, "Stop! You're getting distracted."

Step 2: The "Focus Filter" (Re-Speak)
If the model is confused, the system intervenes. It takes the video and cuts out everything except the specific time interval the GPS signal pointed to.

Hard Crop: It physically cuts the video clip to just those few seconds and asks the model to look at it again.
Soft Mask: It keeps the whole video but puts a "blindfold" over the distracting parts so the model can only "see" the relevant seconds.

Now, with the noise removed, the model looks at the video again. Because the distractions are gone, it can't get confused. It re-reads the question and gives a much more accurate answer.

The Results: "Magic" Without Training

The paper tested this on several different AI models (like Qwen3-VL and MiMo-VL).

No New Training: They didn't have to teach the models anything new. They just changed how the models were asked to answer.
Better Accuracy: The models became significantly better at finding the right time. For example, on one test, the accuracy jumped by 3.5 points, which is a huge deal in this field.
General Use: It worked on general-purpose models (which aren't usually good at this) and even on models that were already trained specifically for this task.

Summary

The paper proves that AI models often have the right answer hidden inside them, but they lose it because the video is too noisy and distracting. By using a "Read, Then Re-Speak" strategy—checking the model's internal focus first, then removing the distractions—the researchers helped the models remember what they already knew, making them much better at telling us when things happen.

Technical Summary: MLLMs Know When Before Speaking

Problem Statement

Video Temporal Grounding (VTG) requires Multimodal Large Language Models (MLLMs) to localize the start and end times of a queried event within an untrimmed video. While modern MLLMs demonstrate fluency in describing video content, their ability to predict precise timestamps remains unreliable. Existing solutions face a trade-off: post-training methods improve performance but require costly temporal annotations and specialized pipelines, while training-free alternatives often rely on coarse heuristics (e.g., segment-caption matching or OCR-style frame reading) that sacrifice boundary precision. This work investigates whether MLLMs genuinely lack temporal grounding capabilities or if they possess a latent ability that standard autoregressive decoding fails to utilize.

Methodology

The authors propose an inference-time "read-then-regenerate" framework that leverages internal attention cues to recover hidden temporal signals without modifying model parameters or architecture.

1. Identification of Temporal Grounding Heads (TG-Heads)

Through a systematic "attention knockout" study, the authors identify a sparse subset of attention heads, termed Temporal Grounding Heads (TG-Heads).

Mechanism: By suppressing the cross-modal attention flow (query-to-video) of individual heads and measuring the drop in VTG accuracy (Grounding Contribution Score, GCS), they found that only a small, heavy-tailed subset of heads is critical for temporal localization.
Finding: Removing these specific heads causes a disproportionate degradation in performance, indicating that MLLMs contain a specialized, latent mechanism for temporal grounding concentrated in these heads.

2. The Perception-Generation Gap

The study reveals a critical disconnect between the model's internal state and its final output:

Prefill Stage: During the initial processing of the query, TG-Heads concentrate attention precisely on the ground-truth interval.
Decoding Stage: As the model autoregressively generates answer tokens, attention shifts away from the correct interval toward visually salient but query-irrelevant segments (distractors) or attention sinks (e.g., the start of the video).
Hypothesis: The model "knows" the correct time during the reading phase but "forgets" it during the speaking phase due to the dilution of the signal by irrelevant visual tokens and the lack of semantic guidance from numeric timestamp tokens.

3. Inference-Time Read-Then-Regenerate Framework

To bridge this gap, the authors propose a two-stage framework:

Stage 1: Initial Inference & Signal Extraction

The MLLM performs a standard forward pass to generate a baseline prediction.
Simultaneously, the cross-modal attention of the identified TG-Heads is extracted during the prefill stage.
Signal Processing:
- Entropy-Based Aggregation: Query tokens are weighted by the negative exponential of their attention entropy. Low-entropy (content-rich) tokens are up-weighted, while high-entropy (function words) tokens are down-weighted.
- Contrastive Debiasing: To remove architectural biases (e.g., attention sinks at the video start), the model processes a "blank video" (zero pixels) with the same query. The resulting "zero-video" attention curve is subtracted from the original signal to isolate query-specific relevance.
Interval Detection: The debiased signal is thresholded, dilated to bridge gaps, and expanded to define a high-attention temporal interval $[f_s, f_e]$ .
Confidence Gate: If the baseline prediction is already confident (based on decode probability) or aligns with the attention peak, Stage 2 is skipped.

Stage 2: Attention-Guided Re-Inference
If the confidence gate triggers, the MLLM is re-invoked with visual context restricted to the detected high-attention interval. Two strategies are employed:

Hard Crop: The video is cropped to the interval, and the model re-reads the segment. This increases per-frame resolution for the target event.
Soft Mask: The original video timeline is preserved, but the attention mask is surgically modified to make out-of-interval visual tokens invisible (attention weight set to $-\infty$ ). This suppresses distractors while maintaining temporal context.

Key Contributions

Discovery of TG-Heads: The identification of a sparse set of attention heads that provide intervention-backed evidence of latent temporal localization within general-purpose MLLMs.
Revealing the Perception-Generation Gap: Demonstrating that MLLMs possess accurate temporal signals during the prefill stage but lose them during autoregressive decoding due to distraction by irrelevant frames.
Training-Free Framework: A plug-and-play inference-time framework that recovers these hidden signals and improves VTG performance across multiple models without parameter updates or architectural changes.

Experimental Results

The framework was evaluated on three benchmarks (QVHighlights-TimeLens, Charades-TimeLens, ActivityNet-TimeLens) using three MLLMs: MiMo-VL-7B, Qwen3-VL-8B, and the VTG-tuned TimeLens-8B.

Performance Gains: The method consistently improved performance across all models and datasets.
- MiMo-VL-7B: Achieved up to +3.5 mIoU improvement on QVHighlights-TimeLens.
- Qwen3-VL-8B: Achieved up to +2.5 mIoU improvement.
- TimeLens-8B: Even the specialized, post-trained model saw gains (e.g., +0.4 mIoU), suggesting the framework corrects residual errors even in fine-tuned models.
Generalization: The approach worked effectively across different model scales (4B to 32B) and architectures.
Efficiency: While the worst-case scenario involves three forward passes (baseline, zero-video, re-inference), the confidence gate skips Stage 2 for the majority of samples (approx. 80–90%), keeping effective overhead manageable.

Significance and Claims

The paper claims that general-purpose MLLMs already possess a latent ability for temporal grounding that is simply obscured by the decoding process. By "reading" the internal attention cues before they are lost and "regenerating" the answer in a context where distractors are suppressed, the method surfaces this capability without the need for expensive retraining.

The authors position this work as a step toward more reliable multimodal reasoning, suggesting that the gap between what a model attends to and what it generates is a fundamental issue that can be mitigated through inference-time interventions. They explicitly state that their method does not invent new capabilities but rather recovers existing, latent ones, offering a path to fine-grained temporal localization that training-free heuristics previously failed to achieve.