💬 NLP

Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs

This paper diagnoses the poor temporal reasoning capabilities of Video-LLMs by identifying bottlenecks in both visual encoders and projector designs, leading to a new architecture that achieves near-perfect performance on the Arrow-of-Time task and significantly improves broader temporal reasoning benchmarks.

Original authors: Peitao Han, Fei Cheng, Lis K. Pereira, Qianying Liu, Shigeru Kitazawa

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peitao Han, Fei Cheng, Lis K. Pereira, Qianying Liu, Shigeru Kitazawa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a video of a glass falling off a table and shattering. If you play that video backward, you see the shards magically flying up and reassembling into a perfect glass. Even a toddler knows this looks wrong. Humans have an innate "time sense" that tells us which way time is flowing.

This paper investigates why advanced AI models, called Video-LLMs (Video Large Language Models), are terrible at this simple task. While humans get it right almost 100% of the time, these AI models often guess randomly, like a coin flip.

The authors decided to play detective to find out where the AI is losing track of time. They broke the AI down into three main parts and traced the "arrow of time" through each one.

Here is the story of their discovery, using simple analogies:

1. The Camera (The Vision Encoder)

First, they looked at the part of the AI that "sees" the video. They found two types of cameras:

The "Frame-by-Frame" Camera: This looks at each picture in the video individually, like flipping through a photo album. It misses the motion between the photos. The paper found these cameras are blind to time. They can't tell if the glass is falling or flying up.
The "Movie" Camera: This looks at the whole video clip at once, understanding how the frames connect. These cameras can see the arrow of time. When the authors tested just this part of the AI (without the rest of the brain), it got the answer right 85–90% of the time.

The Problem: The AI has a good "movie camera," but something goes wrong before the information reaches the brain.

2. The Translator (The Projector)

The video camera speaks a different language than the AI's "brain" (the Language Model). A middleman called a Projector translates the video into text-like tokens the brain can understand.

The authors tested two types of translators:

The "Summarizer" (Q-Former): This translator tries to be efficient. It looks at the whole video and squishes it down into a few key sentences, like a movie review. Unfortunately, in doing so, it throws away the timeline. It tells the brain what happened, but not when or in what order. The paper found this translator destroys the time information, causing the AI to fail.
The "Time-Preserving" Translator (MLP): This translator is careful. It keeps the sequence of events intact, like a script that lists every scene in order. When they used this translator, the AI's performance skyrocketed.

The Discovery: The bottleneck wasn't the camera; it was the translator. The standard translator was accidentally deleting the "time" part of the message.

3. The Brain (The LLM)

Finally, they looked at the brain itself. They found that the brain is actually quite good at understanding time if it receives the information correctly.

However, they noticed that when the camera is trained to talk to the brain (a process called "language alignment"), the camera starts to forget the specific details of time to focus on matching words. It's like a camera that learns to describe a glass so well that it forgets whether the glass is falling or rising.
The brain works best when it gets raw, unfiltered time data from the earlier layers of the camera, passed through the "Time-Preserving" translator.

The Solution: Building a Time-Savvy AI

Using these clues, the authors built a new Video-LLM with three specific upgrades:

A "Movie" Camera: One that explicitly understands motion.
A "Time-Preserving" Translator: One that refuses to squish the timeline.
Special Training: They taught the AI specifically on the "Arrow of Time" task (forward vs. backward videos).

The Result:
This new AI didn't just get better; it beat humans. It achieved 98.1% accuracy on the time-direction test, compared to the human average of 89.2%.

Furthermore, this training didn't just help with the time test. It made the AI better at other tasks that require understanding time, such as figuring out the order of events in a story or the direction of movement, improving its scores on other benchmarks by up to 6 points.

Summary

The paper concludes that for an AI to understand time, it needs two things:

A camera that actually records time, not just pictures.
A translator that doesn't delete the timeline when passing the message to the brain.

Once they fixed the "leak" in the translator, the AI could finally see the arrow of time clearly.

Technical Summary: Tracing the Arrow of Time in Video-LLMs

Problem Statement

Despite humans solving the Arrow-of-Time (AoT) task—determining whether a video plays forward or backward based on physical irreversibility—with near-perfect accuracy, frontier Video Large Language Models (Video-LLMs) perform only marginally above chance. This performance gap raises a critical diagnostic question: Is the failure due to visual backbones' inability to encode temporal information, or does the bottleneck lie elsewhere in the Video-LLM architecture (e.g., the projector or the LLM)? Current Video-LLMs, which typically couple a vision encoder with an LLM via a projector, struggle with temporal reasoning, a prerequisite for understanding real-world dynamics.

Methodology

The authors employ a systematic, component-wise tracing approach to isolate where temporal information is encoded, preserved, or lost within the Video-LLM pipeline. The study is structured around three Research Questions (RQs):

1. RQ1: Can Temporal Information Be Encoded by the Vision Encoder?

Approach: The authors decouple the vision encoder from the LLM and employ an "attentive probing" technique. A lightweight, attention-based classifier is trained on top of frozen vision encoders to predict video direction (forward/backward).
Encoder Comparison: They compare frame-centric encoders (which process videos as individual frames and rely on positional embeddings for temporal cues, e.g., Qwen2VL, DINOv2) against video-centric encoders (which explicitly model temporal dynamics across frames, e.g., VJepa, InternVideo2).
Finding: Video-centric encoders with explicit temporal modeling successfully encode strong AoT signals (achieving ~~85-88% accuracy in probing). In contrast, frame-centric encoders fail catastrophically, remaining near chance level (~~50%).

2. RQ2: How Can Temporal Information Be Effectively Transferred to LLMs?

Approach: The authors investigate the "information bottleneck" by analyzing the projector (the component mapping visual features to LLM tokens) and the layer-wise dynamics of the encoder.
Projector Analysis: They compare three projector designs:
- Q-Former: Uses learnable query tokens to attend to visual representations.
- Time-Compressed MLP: Pools features along the temporal dimension before projection.
- Time-Preserved MLP: Preserves the temporal dimension ( $T'$ ) while pooling only spatial dimensions ( $H' \times W'$ ).
Layer-wise Analysis: They trace representations from different layers of the encoder (specifically InternVideo2 Stage 1 vs. Stage 2) through the projector to the LLM.
Findings:
- Projector Bottleneck: Q-Former disrupts temporal information, causing performance to collapse to chance levels even when using strong video-centric encoders. Time-preserved MLPs significantly outperform Q-Former and time-compressed MLPs, indicating that preserving temporal resolution during projection is critical.
- Language Alignment Impact: While Stage 1 encoders (pre-trained on video only) preserve explicit temporal information in early-to-mid layers, Stage 2 encoders (aligned with language via CLIP-style training) show a degradation in temporal transferability. The temporal information in Stage 2 becomes less explicit and harder for the LLM to decode, even with a time-preserved projector.

3. RQ3: Can AoT Supervision Improve Broader Temporal Reasoning?

Approach: The authors construct a Video-LLM combining a temporal-aware video encoder (InternVideo2 Stage 1), a time-preserved MLP projector, and an LLM. They fine-tune this model with AoT supervision (forward/backward classification) alongside general instruction tuning.
Evaluation: They test the model on the AoT benchmark (AoTP P B) and broader temporal reasoning benchmarks (VITATECS-Direction, TVBench), comparing against general video benchmarks (MVBench, Video-MME) to ensure no degradation in general capabilities.

Key Contributions

Diagnosis of Temporal Encoding: Demonstrated that explicit temporal modeling in video-centric encoders is sufficient to encode strong AoT signals, whereas frame-centric encoders are fundamentally insufficient for this task.
Identification of Architectural Bottlenecks: Identified the projector design as a primary bottleneck. Specifically, Q-Former architectures disrupt temporal flow, while time-preserved MLPs are essential for transferring temporal information to the LLM.
Impact of Language Alignment: Revealed that CLIP-style language alignment (Stage 2 training) makes temporal information less explicitly accessible and harder to transfer to the LLM compared to pure video-centric pre-training (Stage 1).
Performance Breakthrough: Built a Video-LLM that surpasses human performance on the AoTP P B benchmark (98.1% accuracy vs. human 89.2%).
Generalization: Showed that AoT supervision acts as an effective training signal that improves broader temporal reasoning tasks (up to +6.0 points on VITATECS-Direction) without harming general video understanding.

Results

Probing: Video-centric encoders (e.g., VJepa-H) achieved 88.8% accuracy on AoT probing, while frame-centric encoders (e.g., Qwen2.5VLViT) hovered at 50.7%.
VQA Performance: Standard Video-LLMs (e.g., InternVideo2VL with Q-Former) performed at chance level (50.0%) on AoT VQA despite using a strong video-centric encoder.
Projector Ablation: Replacing Q-Former with a time-preserved MLP on InternVideo2 Stage 1 boosted AoT VQA accuracy from 50.0% to 95.1%.
Final Model: The proposed model (Stage 1 encoder + Time-Preserved MLP + AoT supervision) achieved 98.1% on AoTP P B.
Broader Tasks: The model improved VITATECS-Direction by 6.0 points and TVBench by 1.3 points. Performance on general benchmarks (MVBench, Video-MME) remained stable, indicating the gains are specific to temporal reasoning.

Significance

The paper argues that the failure of current Video-LLMs in temporal reasoning is not solely due to a lack of temporal encoding in the visual backbone, but rather a failure in information transfer caused by inappropriate projector designs and the side effects of language alignment. By tracing the "arrow of time" through the architecture, the authors provide a mechanistic explanation for these failures and propose practical architectural principles (video-centric encoders + time-preserved projectors) to overcome them. The work suggests that explicit supervision on physical irreversibility (AoT) is a viable and effective strategy to enhance the temporal reasoning capabilities of multimodal models.