💻 computer science

Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No

This paper introduces FV-Action, a training-free temporal grounding method that significantly outperforms existing models by replacing direct timestamp regression with a coarse-to-fine binary questioning strategy, thereby revealing that VLM failures stem from the task interface rather than perceptual limitations.

Original authors: Ji Huang, Barry Devereux, Hui Wang

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ji Huang, Barry Devereux, Hui Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can watch a video and tell you exactly what is happening in it. If you ask, "Is the dog chasing the ball?" it will say "Yes!" with perfect confidence. This is the world of Multimodal Large Language Models (VLMs): computers that can see images and read text, combining them to understand the world. But there is a tricky part of this job called Temporal Video Grounding. It's not just about knowing what is happening; it's about knowing when it happens. You need to point to the exact start and end seconds of an event in a long, uncut video. Think of it like finding a specific song in a three-hour mixtape just by humming a few notes. Scientists have been trying to teach these robots to do this by asking them to spit out numbers like "start at 12.5 seconds, end at 18.2 seconds." But for a long time, even the smartest robots have been terrible at this, often guessing wildly or getting the numbers completely wrong while sounding very sure of themselves.

This paper, titled "Your VLM Already Knows When," investigates why these robots are failing at telling time and discovers a surprisingly simple fix. The authors found that the problem isn't that the robot is "blind" or doesn't understand the video. The problem is the question they are being asked. Asking a robot to "regress" (calculate) a specific decimal number is like asking a human to guess the exact temperature of a cup of coffee without a thermometer; they might guess a plausible number, but they are just guessing. Instead, the researchers found that if they ask the robot a simple "Yes or No" question about small clips of the video ("Does this clip show the dog chasing the ball?"), the robot becomes incredibly accurate at spotting the right moment. By scanning the video with these yes/no questions and picking the best spot, they boosted the robot's accuracy from a dismal 3.8% to a massive 56.8% on one major test, all without teaching the robot anything new.

The Mystery of the Confidently Wrong Robot

The story starts with a frustrating observation. When researchers asked powerful video models to tell them when an event happened, the models would often give answers that were completely wrong. But here's the weird part: the models were confidently wrong. In the world of AI, we usually think that if a model is unsure, it might make a mistake. But these models had very low "uncertainty" (a technical way of saying they were very sure) even when they were wrong. It was as if a student taking a math test was 100% certain that 2+2=5. The researchers tested whether they could use the model's own "confidence" to catch these errors, but it didn't work. The models were so sure of their wrong answers that no amount of "confidence checking" could separate the right answers from the wrong ones.

This led the authors to a big realization: the robot wasn't failing because it couldn't see the event. It was failing because of how it was being asked to answer.

The Magic Switch: From "Guessing Numbers" to "Playing Yes/No"

The paper argues that the failure lies in the interface—the way we talk to the robot.

The Old Way (Regression): We ask the robot to generate decimal timestamps (e.g., "12.45 seconds"). The problem is that these specific numbers don't appear often in the data the robot learned from. So, when asked, the robot falls back on guessing plausible-sounding numbers based on language patterns, not visual evidence. It's like asking someone to guess the exact weight of a mystery box; they might say "5.2 pounds" because it sounds like a reasonable guess, not because they actually weighed it.
The New Way (Binary VQA): The researchers tried a different approach. Instead of asking for a number, they asked the robot a simple Yes or No question about short clips of the video: "Does this clip show the event?"

They built a system called FV-Action that works like a detective scanning a crime scene.

Coarse Scan: The system takes a wide look at the video, breaking it into chunks and asking the robot, "Is the event happening in this chunk?"
Fine Scan: Once it finds a chunk that gets a "Yes," it zooms in on that area and asks the same question again, but with smaller, more precise chunks.
The Winner: It doesn't care about the exact "Yes" score (like 0.85 vs 0.86); it just cares about ranking. Which chunk got the strongest "Yes"? That's where the event is.

The results were shocking. When they switched from asking for numbers to asking Yes/No questions, the accuracy jumped massively. For one model (InternVL2-8B), the score went from 3.8% to 53.4%. For another (Qwen2.5-VL-7B), it went from 28.4% to 56.8%. This happened even though the robot's "brain" (the model weights) didn't change at all. They didn't train the robot on new data; they just changed the game they were playing.

Two Reasons Why It Still Isn't Perfect

Even with this huge improvement, the system isn't perfect. The authors broke down the remaining mistakes into two distinct categories, like two different gears in a machine:

The Perception Axis (The "Eyes"): Sometimes the robot just doesn't see the event clearly. If the video is blurry or the action is very subtle, the robot might not be able to tell the difference between the right clip and a wrong one. This depends entirely on how good the robot's "vision" is. If you swap in a smarter robot model, this part gets better automatically.
The Geometry Axis (The "Ruler"): This is a math problem, not a vision problem. The system picks a spot and draws a fixed-size window around it (like a 20-second box). If the event you are looking for is only 5 seconds long, but your box is 20 seconds, the box will be too big, and the math says you failed, even if you found the right spot. The authors showed that this failure is predictable. If you know the size of the event and the size of your box, you can calculate exactly how often you will fail, just by doing a simple division.

The Takeaway

The paper concludes that the "magic" wasn't in making the robot smarter; it was in asking the right question. By stopping the robot from trying to guess numbers it doesn't know how to generate, and instead letting it use its natural ability to judge "Yes or No," the researchers unlocked a hidden talent the robot already had.

They also showed that you don't need to train a robot for years to get good at this. You just need to ask it to play a game of "Hot or Cold" with Yes/No questions. While the system still struggles with very short events in long videos (because of the "Geometry" problem), it proves that the biggest barrier to robots understanding when things happen isn't a lack of knowledge—it's just a bad interface. And sometimes, the best way to fix a super-intelligent robot is to stop asking it to do math and start asking it to play a simple game.

Technical Summary: Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No

1. Problem Statement

Temporal Video Grounding (TVG) requires locating the specific start and end timestamps $[s, e]$ of an event described by a natural language query within an untrimmed video. While Multimodal Large Language Models (MLLMs) excel at visual reasoning, they systematically fail when tasked with directly regressing decimal timestamps.

The paper identifies a critical failure mode:

Confident Hallucination: Strong VLMs (e.g., InternVL2-8B) achieve extremely low accuracy (e.g., 3.8% R@0.5 on Charades-STA) when prompted for timestamps. Crucially, 77–80% of these errors are "confident," characterized by low output entropy.
Ineffective Uncertainty Detection: Because the models are confidently wrong, entropy-based error detection mechanisms perform worse than a random classifier (F1 < 0.36).
The Bottleneck: The failure is not a lack of perceptual capability (the models can "see" the event) but a format mismatch. Pre-training corpora lack the specific distribution of language-conditioned decimal timestamps, causing models to fall back on linguistic priors and generate plausible but ungrounded values.

2. Methodology: FV-Action

The authors propose FV-Action, a training-free framework that bypasses timestamp regression entirely. Instead of asking the model to generate numbers, it asks the model to judge binary questions, leveraging the model's existing discriminative capabilities.

Core Mechanism

The framework treats the frozen VLM as a measurement instrument that answers the binary question: "Does this clip show {query}?"

The pipeline operates in three stages:

Coarse Scanning:
- The video is divided into $K_c$ clips with uniformly spaced centers.
- Each clip is represented by $F=5$ frames (symmetrically extracted).
- The frozen backbone answers the binary VQA for each clip.
- The system reads the probability $P(\text{Yes})$ from the logits of the first generated token (Yes/No surface forms).
- Key Insight: The system relies on the ranking of these probabilities across clips within a video, not their absolute calibrated values. This allows poorly calibrated models to function effectively as scanners.
Peak Localization:
- Non-maximum suppression (NMS) selects the top $N_p$ candidate peaks from the coarse probability curve.
- A fine scan re-examines the neighborhood of each candidate with $K_f$ additional clips.
- A joint score (geometric mean of coarse and fine scores) selects the final peak $t^*$ .
Window Generation:
- The final output is a fixed-width window $W$ centered on $t^*$ .
- $W$ is a domain prior (e.g., 8s for Charades-STA, 80s for ActivityNet) derived from training set statistics, not learned.

Design Rationale

No Training: The backbone remains frozen; no temporal supervision or fine-tuning is used.
Ranking over Calibration: The method exploits the model's ability to order clips correctly (discriminability) rather than requiring it to output precise probabilities.
Uniform Sampling: Probes are placed uniformly rather than adaptively, as visual salience does not reliably correlate with action onset.

3. Key Contributions

A. Diagnostic Study on Temporal Hallucinations

The paper establishes that MLLM temporal failures are systematic format hallucinations, not epistemic uncertainty.

Finding: 77–80% of wrong predictions have low entropy.
Implication: Confidence thresholds cannot separate correct from incorrect predictions; the failure is inherent to the generative interface.

B. The Interface Bottleneck

Through controlled experiments with identical frozen backbones and inputs, the authors prove the interface is the limiting factor.

Result: Replacing timestamp regression with binary VQA scanning improves R@0.5 by 28 to 50 percentage points across four backbones (InternVL2, Qwen2.5-VL, LLaVA-NeXT, mPLUG-Owl3).
Example: InternVL2-8B jumps from 3.76% to 53.39% R@0.5 on Charades-STA solely by changing the output format.

C. Two-Axis Failure Decomposition

Residual errors are decomposed into two separable axes:

Perception Axis: Governed by the backbone's per-clip discriminability. This can be measured pre-deployment via an "oracle gap" (difference in $P(\text{Yes})$ between ground-truth and negative clips). Stronger backbones yield better grounding.
Geometry Axis: Governed by the fit between the fixed output window $W$ $W$ and the event width $g$ $g$ .
- Analytic Predictability: Accuracy is analytically predictable by the ratio $\min(W, g) / \max(W, g)$ . If the event falls outside the feasible band $[W/2, 2W]$ , R@0.5 is theoretically zero regardless of peak localization quality.
- Implication: Post-processing (smoothing, contrastive correction) cannot fix geometry errors; only adjusting the window size or the backbone can.

D. FV-Action Performance

Charades-STA: Achieves 56.8% R@0.5, surpassing the same backbone's native temporal pipeline (53.6%) and all prior training-free methods.
Zero-Shot Generalization: Outperforms trained models on TACoS and improves over direct prediction on ActivityNet Captions and QVHighlights without any temporal supervision.

4. Experimental Results

Dataset	Metric	FV-Action (Qwen2.5-VL)	Best Prior Training-Free	Best Supervised (for context)
Charades-STA	R@0.5	56.8%	40.0% (BTDP)	63.2% (OmniVTG)
ActivityNet	R@0.5	26.7%	34.0% (GranAlign)*	50.7% (TimeMarker)
TACoS	R@0.5	13.0%	N/A (Zero-shot)	35.0% (UniVTG)
QVHighlights	R1@0.5	43.0%	61.9% (GranAlign)*	53.9% (Moment-DETR)

*Note: GranAlign and BTDP use multi-model pipelines (captioning + LLM alignment), whereas FV-Action uses a single VLM.

Key Observations:

Backbone Independence: The interface gain (+28 to +50 pp) holds across all tested architectures.
Geometry Limitation: On ActivityNet, where event lengths vary wildly, the fixed window causes structural failures on short events. An adaptive window variant helps short events but hurts long ones, resulting in a net neutral gain, confirming the geometry axis is a hard constraint for fixed-window methods.
Cost: The method trades training for inference cost (36–120 forward passes per sample vs. 1 for regression), but requires no labels or gradient updates.

5. Significance and Claims

The paper claims that the "grounding ability" already exists within frozen VLMs but is buried by the generative interface.

Paradigm Shift: It demonstrates that TVG does not require the model to act as a numerical regressor. By reframing the task as a binary discrimination problem, the model's latent temporal understanding is unlocked.
Diagnostic Utility: The two-axis analysis provides a prescriptive tool for practitioners. Before deployment, one can measure the "perception axis" (oracle gap) to determine if a stronger backbone is needed, or analyze the "geometry axis" (event width distribution) to determine if the output window rule needs adjustment.
Modest Scope: The authors acknowledge limitations: the method relies on test-time computation, struggles with compound queries (multiple events), and cannot fully resolve the geometry mismatch in datasets with highly heterogeneous event durations without adaptive windowing (which they show has trade-offs).

In summary, the paper argues that the interface, not perception, is the bottleneck for temporal grounding in VLMs, and that a simple, training-free binary scanning strategy can recover the majority of the performance gap between general-purpose VLMs and specialized supervised systems.