Technical Summary: What's the Catch? Evaluating Temporal Consistency in Vision-Language Models

Problem Statement

Vision-language models (VLMs) have demonstrated strong performance on video and image-sequence benchmarks. However, it remains unclear whether these models genuinely capture temporal structure or if their success stems from exploiting shortcuts, such as relying on static frame content, textual biases, or small subsets of frames. Existing benchmarks often rely on question-answering or captioning tasks, which make it difficult to distinguish between true temporal reasoning and the ability to solve tasks without understanding event ordering. Consequently, high benchmark performance does not necessarily indicate that a model can detect violations of temporal consistency.

Methodology: TimeCatch Benchmark

To address this ambiguity, the authors introduce TimeCatch, a controlled benchmark designed to evaluate temporal grounding through anomaly detection. The framework isolates temporal reasoning by introducing specific perturbations while keeping visual content constant.

Task Formulation

The benchmark distinguishes between two types of anomalies to disentangle temporal reasoning from frame-level perception:

1. Temporal Anomalies: Created by swapping consecutive frames within a sequence. Detecting these requires reasoning about the plausibility of event progression over time.
2. Frame-Level Anomalies: Created by replacing a single frame with Gaussian noise. These serve as a control condition; detecting them requires only attention to individual frames without temporal context.

The evaluation consists of four specific tasks across both anomaly types:

• Detection: Determining whether a sequence contains an anomaly.
• Localization: Identifying the specific location (frame index) of the anomaly.

Dataset Curation

TimeCatch utilizes four datasets spanning synthetic and real-world domains:

• Synthetic: CLEVRER (3D object interactions) and CRAFT (2D object interactions).
• Real-world: DriveLM (driving scenarios) and MTL-AQA (competitive diving).

Sequences are filtered to ensure consecutive frames exhibit distinguishable changes (using LPIPS metrics) and to remove sequences shorter than four frames. The benchmark includes sequences of varying lengths (4–16 frames) to analyze the effect of context length.

Experimental Setup

The study evaluates five state-of-the-art open-weight VLMs (Qwen2.5-VL-7B, Qwen3-VL-8B, Gemma-4-E4B, InternVL3-8B, InternVL3.5-8B) in a zero-shot setting. A human study was also conducted to establish a performance baseline. Models were evaluated using accuracy for both detection (binary classification) and localization (correct index prediction).

Key Results

1. The Temporal Reasoning Gap

The primary finding is a substantial performance gap between frame-level and temporal anomaly detection:

• Frame-Level Performance: VLMs consistently detect and localize frame-level anomalies with high accuracy (reaching up to 99.6% detection and 99.4% localization). This confirms that models can attend to image sequences and identify corrupted frames.
• Temporal Performance: In contrast, VLMs perform near chance on temporal anomaly detection (highest accuracy ~57.4%) and only modestly above chance on localization (highest accuracy ~43.2%).
• Human Comparison: Humans achieve near-ceiling performance on both tasks (75–91% detection, 83–89% localization), highlighting that the task difficulty is not inherent but specific to current VLM architectures.

2. Analysis of Failure Modes

The authors conducted several analyses to determine the causes of these failures:

• Visual Similarity: Performance did not significantly improve even when swapped frames were perceptually distinct (high LPIPS distance), suggesting the issue is not low-level perception.
• Localization Bias: Confusion matrices revealed that models do not merely "miss" the correct location; they exhibit a strong systematic bias toward predicting specific positions (e.g., position 2) regardless of the true anomaly location.
• Model Scaling: Increasing model capacity (from 2B to 32B parameters) improved frame-level performance to near-ceiling levels but yielded only marginal gains on temporal tasks.
• Prompting and Context: Variations in prompting strategies, the removal of scene descriptions, and enabling "thinking" modes produced no consistent improvement in temporal reasoning.
• Sequence Length: Increasing sequence length (up to 16 frames) did not improve temporal anomaly detection, whereas frame-level localization degraded as the number of candidate frames increased.

Key Contributions

1. Systematic Evaluation: A comprehensive assessment of state-of-the-art VLMs and human performance on temporal anomaly detection and localization.
2. Evidence of a Gap: Demonstration that current models can identify frame-level anomalies but struggle significantly to reason about temporal consistency across image sequences.
3. TimeCatch Benchmark: The introduction of a controlled benchmark that isolates temporal grounding through anomaly detection, providing a simple and direct test for temporal consistency.

Significance and Implications

The paper argues that strong performance on existing video benchmarks may overestimate the temporal reasoning capabilities of VLMs. The findings suggest that current models reason effectively about individual frames but fail to integrate information across frames to reason about temporal consistency.

This limitation persists regardless of visual similarity, prompting strategies, model scale, or sequence length, indicating a fundamental challenge in modeling temporal relationships rather than a limitation of specific evaluation settings. The authors posit that temporal anomaly detection serves as a necessary complementary evaluation setting for tracking progress toward truly temporally grounded VLMs, particularly for applications like autonomous driving and robotics where decisions depend on the evolution of events over time.

The authors view TimeCatch as a diagnostic tool to isolate a fundamental capability gap, rather than a comprehensive evaluation of all forms of temporal reasoning (such as long-range dependencies or causal structures), which remain areas for future work.