💻 computer science

Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning

This paper introduces Know-Show, a new benchmark and a training-free method called GRAM, to evaluate and improve the spatio-temporal grounded reasoning capabilities of Video-Language Models by unifying reasoning with visual and temporal localization across diverse scenarios.

Original authors: Chinthani Sugandhika, Chen Li, Deepu Rajan, Basura Fernando

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chinthani Sugandhika, Chen Li, Deepu Rajan, Basura Fernando

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a home security video. A person walks into the kitchen, opens the fridge, takes out a carton of milk, and pours it into a glass.

If you ask a human, "Who poured the milk and when did they do it?", they can instantly point to the person, say "It was the guy in the blue shirt," and pinpoint the exact moment on the clock. They know the answer and can show you the proof.

Now, imagine asking a super-smart AI robot the same question. The paper you shared, titled "Know-Show," reveals a funny but serious problem: The robot might say, "A person poured milk!" (The Know part is okay). But if you ask, "Show me exactly who and when," the robot might point to the wrong person, guess the wrong time, or just stare blankly. It knows the story, but it can't ground that story in the actual video evidence.

Here is a simple breakdown of what the researchers did to fix this.

1. The Problem: The "Hallucinating" Detective

Current AI video models are like detectives who read a mystery novel but have never seen the crime scene. They are great at guessing the plot ("The butler did it!") because they've read millions of stories. But when you ask them to point to the specific moment the butler picked up the gun, they get lost.

The Gap: They can recognize objects (a fridge, a hand) and understand time (first this, then that), but they struggle to link them together. They can't say, "The left hand of Person A touched the fridge handle at 12:05 PM."
The Result: In the real world (like self-driving cars or medical monitoring), this is dangerous. If a car AI thinks a pedestrian is crossing but can't point to where or when, it might crash.

2. The Solution: The "Know-Show" Benchmark

The researchers created a new test called Know-Show. Think of this as a final exam for video AI, but with a twist.

Instead of just asking, "What happened?", the exam asks:

Know: What is happening?
Show: Point to the exact person, object, or hand involved, and tell me the exact second it happened.

The test has five levels of difficulty, like a video game:

Level 1: Point to the person doing the action.
Level 2: Point to the object being used.
Level 3: Point to both the person and the object together.
Level 4 (The Boss Level): Point to the specific hand touching the object. (This is very hard because hands move fast and are small).
Level 5: Tell me the exact order of events and when they started.

The Score: The AI only gets points if it gets the answer and points to the right spot at the right time. If it gets the answer right but points to the wrong person, it gets zero.

3. The Results: AI vs. Humans

When they ran this test on the smartest AI models available (like GPT-4o, Gemini, and others), the results were humbling:

Humans: Scored over 75%. We naturally connect actions to specific people and times.
AI Models: Scored very low (often under 20-30%). They were great at guessing the general idea but terrible at the "proof." They often confused who was doing what, or missed the exact moment an action started.

4. The Fix: GRAM (The "Flashlight" Plugin)

The researchers didn't want to retrain the massive AI models (which takes years and millions of dollars). Instead, they built a "plug-in" called GRAM.

Think of GRAM as a flashlight and a time-stamp marker for the AI:

The Flashlight (Attention Selection): When the AI starts to reason ("Okay, who opened the fridge?"), GRAM forces the AI to look at the specific video frames where the fridge is actually being opened. It filters out the blurry background and focuses only on the relevant pixels.
The Time-Stamp Marker: The AI usually guesses time based on a vague feeling. GRAM literally writes the time (e.g., "5.2 seconds") into the AI's thought process, forcing it to align its answer with the clock on the video.

The Outcome: When they added this "flashlight" to the AI, its performance jumped significantly. It didn't become perfect, but it finally started to "show what it knows" instead of just guessing.

The Big Picture

This paper is a wake-up call. It tells us that while AI is getting better at "talking" about videos, it still struggles to "see" and "prove" what it's talking about.

Before: AI was like a storyteller who makes up details.
After (with Know-Show & GRAM): We are teaching AI to be a forensic analyst who must back up every claim with visual evidence and a timestamp.

This is crucial for the future. If we want robots to help in hospitals, drive cars, or monitor safety, they can't just guess; they need to know exactly who did what and when, and be able to show us the proof.

1. Problem Statement

Current Large Video-Language Models (Video-LMs) have made significant strides in multimodal understanding but suffer from a critical limitation: their reasoning is weakly grounded in space and time.

The Gap: Existing models can often recognize objects or the coarse temporal order of actions, but they fail to simultaneously reason about an action's semantics and ground that reasoning in precise visual evidence (bounding boxes) and temporal evidence (specific timestamps).
The "Know-Show" Deficit: Models often "know" the answer (can predict the correct label) but cannot "show" the evidence (locate the specific person, object, or time window in the video that justifies the answer). Conversely, they may localize an entity but infer the wrong action.
Limitations of Existing Benchmarks: Current benchmarks are disjoint. Localization-centric datasets (e.g., Action Genome) focus on detection/recognition without complex reasoning. Reasoning-centric datasets (e.g., MV-Bench, STAR) test logic but do not require the model to point to the specific visual evidence in the video. There is no unified framework to evaluate Spatio-Temporal Grounded Reasoning (STGR).

2. Methodology

A. The Know-Show Benchmark

The authors introduce Know-Show, a new benchmark designed to evaluate STGR by unifying reasoning and localization.

Dataset Construction: Built from Charades, Action Genome, and Ego4D, containing 2,500 high-quality human-authored questions.
Evaluation Framework: The benchmark consists of five complementary scenarios divided into two categories:
1. Action-Conditioned Spatial Grounded Reasoning:
  - Person Grounded (PGR): Link action to the specific person.
  - Object Grounded (OGR): Link action to the specific object.
  - Person-Object Co-Grounded (POCoGR): Link action to both interacting entities simultaneously.
  - Hand-Object Co-Grounded (HOCoGR): Fine-grained reasoning linking actions to specific hands (left/right) and objects (critical for robotics/embodied AI).
2. Action-Conditioned Temporal Grounded Reasoning (TGR):
  - Reasoning about the relative order of actions and their absolute start/end timestamps.
Success Criteria: A model is only credited if it correctly answers the reasoning question AND provides the correct spatial (bounding box) and temporal (timestamp) localization within the action's duration.

B. The GRAM Framework (Grounded Reasoning Augmentation Module)

To address the performance gap, the authors propose GRAM, a training-free, plug-and-play module that enhances existing Video-LMs.

Core Mechanism 1: Attention-Based Video Token Selection (ST-GR):
- During the decoding process, when the model generates a token marking a new reasoning step, GRAM aggregates attention weights across all layers and heads.
- It selects the top- $N$ most attended video tokens (visual evidence) and concatenates them with the decoder's input embeddings.
- Goal: Force the model to ground each reasoning step in the most relevant visual evidence rather than relying on language priors.
Core Mechanism 2: Explicit Timestamp Tokens (ETT):
- Current models rely on implicit positional embeddings (e.g., MRoPE) which are insufficient for absolute temporal localization.
- GRAM interleaves explicit timestamp tokens (e.g., <0.0s>, <5.1s>) with video frame tokens in the input sequence.
- Goal: Provide the model with direct access to absolute time anchors to improve temporal alignment.

3. Key Contributions

Know-Show Benchmark: The first unified benchmark evaluating the "Know-Show" capability (reasoning + grounding) across five distinct scenarios, including the challenging fine-grained hand-object interactions.
Comprehensive Evaluation: A systematic evaluation of 10+ state-of-the-art Video-LMs (including Qwen2.5-VL, VideoR1, GPT-4o, Gemini, etc.) and human baselines.
GRAM Framework: A simple, training-free augmentation that significantly improves spatio-temporal grounding without retraining the base model.
Insights into Model Limitations: Identification of specific failure modes, particularly in co-grounded reasoning (person-object, hand-object) and absolute temporal localization.

4. Experimental Results

Performance Gaps

Human vs. Model: Humans achieve >75% accuracy across all metrics. Current Video-LMs struggle significantly, with many scoring below 20% on complex co-grounded tasks.
Spatial Reasoning:
- Hierarchy of Difficulty: Performance follows the order: Person (easiest) > Object > Person-Object > Hand-Object (hardest).
- Failure Modes: Models often localize a salient person but fail to bind them to the correct action within the time window. Hand-object reasoning is nearly impossible for current models due to the need for fine-grained contact detection and occlusion handling.
Temporal Reasoning:
- Models perform better on relative temporal order (A happens before B) than absolute temporal localization (A happens at 12.5s).
- Even top models (Gemini, GPT-4o) show large Mean Absolute Deviation (MAD) in timestamp prediction, indicating they capture coarse regions rather than precise boundaries.

Impact of GRAM

Improvements: Applying GRAM to Qwen2.5-VL (7B) significantly boosts performance across all categories.
- Spatial: Increases AccSGR from ~6% (base) to 26.6% (PGR) and 21.2% (OGR).
- Temporal: Improves temporal grounding accuracy (AccTGR) significantly.
Ablation: Both components (ST-GR and ETT) contribute independently, but their combination yields the best results, proving that explicit visual grounding and explicit time encoding are both necessary.
Prompting: The paper demonstrates that Chain-of-Thought (CoT) prompting or multi-step prompting alone does not solve the grounding problem; architectural/representation-level changes (like GRAM) are required.

5. Significance and Future Directions

Real-World Applicability: Robust STGR is essential for robotics, embodied AI, and safety-critical surveillance (e.g., verifying a person turned off a stove requires seeing the hand, the knob, and the specific time).
Interpretability: GRAM provides a mechanism for "interpretable reasoning," where the model's logic is explicitly tied to visual evidence, reducing hallucinations.
Future Work: The authors suggest that future progress requires:
- Better pre-training on egocentric and hand-object interaction data.
- Explicit relational modeling of action-role binding.
- Architectural mechanisms for fine-grained spatial supervision under occlusion.
- Moving beyond implicit temporal embeddings to explicit temporal supervision.

In conclusion, Know-Show exposes a fundamental weakness in current Video-LMs: the inability to simultaneously reason and ground. The proposed GRAM module offers a practical, training-free path toward bridging this gap, though significant room for improvement remains to reach human-level performance.