Technical Summary: The Low-Frequency Trap: Video–Language Models Fail at Simple Event Bookkeeping

Problem Statement

Real-world video benchmarks for Video-Language Models (VLMs) often entangle multiple variables—event count, rate, duration, visual complexity, and semantics—making it difficult to isolate specific failure modes. While existing programmatic benchmarks offer better control, they typically score only the final answer, failing to audit whether a model missed events, confused their timing, lost track of a sequence, or erred only during aggregation. Consequently, final-answer accuracy metrics can conceal weak temporal access or brittle reasoning, masking the fact that models may fail at elementary event bookkeeping as temporal load increases.

Methodology

The authors introduce trace-grounded parametric profiling, a controlled evaluation framework designed to isolate temporal reasoning capabilities from visual complexity.

Controlled Task Space

The study utilizes programmatically generated videos using the Manim animation engine to ensure frame-accurate control over spatial trajectories and timing. Three distinct domains are evaluated:

1. Bounce Ball: Counting wall contacts of a bouncing ball.
2. Blinking: Counting on-pulses of a visual object (transient events).
3. State Machine: Counting categorical state transitions (persistent events).

Experimental Parameters

The framework systematically varies two orthogonal parameters while holding rendering, semantics, and task rules fixed:

• Event Count ($N$): Ranging from 0 to 12.
• Event Frequency ($F$): Ranging from 0.5 Hz to 4.0 Hz.
• Duration: All clips are fixed at 24 seconds. The active event sequence spans approximately $N/F$ seconds, with static frames filling the remainder.

Ground Truth and Metrics

Every video is paired with an executable ground-truth trace containing event timestamps, state changes, and cumulative counts. This allows for a granular evaluation beyond simple final-answer accuracy. Key metrics include:

• Final Answer Exact Match (Acc): Whether the reported count equals $N$.
• Trace Precision ($P$) and Recall ($R$): Alignment between model-reported timestamps and ground truth within a rate-relative window $\delta(F) = 1/(2F)$.
• Visual Observation Ratio (VOR): The ratio of reported events to true events ($M/N$), indicating over- or under-reporting.
• Accidental Correctness Rate (ACR): Cases where the final count is correct despite a low-fidelity trace.
• Reasoning Failure Rate (RFR): Cases where the trace is faithful (high F1) but the final count is incorrect.

Models Evaluated

The primary evaluation focuses on Gemini 3.6 Flash (receiving ~1 FPS input) and Qwen3-VL-235B (receiving 2 FPS input). Supplementary cross-system checks include various scales of Qwen3-VL and InternVL3.5.

Key Contributions

1. Controlled Evaluation Framework: Replaces single aggregate scores with capability surfaces mapped over event count ($N$) and frequency ($F$).
2. Trace-Grounded Benchmark: Audits model-reported events against executable ground truth, distinguishing between unsupported correct counts and faithful event recovery.
3. Targeted Interventions: Diagnoses error sources via denser sampling, event-centered keyframes (oracle evidence), and various prompting strategies (e.g., Chain-of-Thought, structured tracing).
4. Natural-Video Transfer: Validates whether failure patterns observed in controlled settings persist in real-world repeated-event videos (TransRAC dataset).

Results

Staged Temporal Failure

The results reveal a "staged" failure pattern dependent on event representation (persistent vs. transient):

• Persistent Events (State Machine): Gemini 3.6 Flash reliably counts up to $N=12$ at 0.5 Hz and 1.0 Hz. Performance degrades as frequency increases, dropping to $N=2$ at 1.5 Hz.
• Transient Events (Blinking): No reliable positive-count region exists; the model fails to consistently access transient events even at low counts and frequencies.
• High-Load Regime: In high-count, high-frequency conditions, only 0.2% of final counts are correct, and the model recovers just 18.1% of true events.

The Role of Visual Access vs. Reasoning

• Sampling Density: Increasing sampling rate from 1 FPS to 4 FPS improved Bounce Ball accuracy from 19.6% to 29.3%. However, the reported sequence agreed with ground truth only 3.7% of the time. This indicates that extra frames can inflate final scores (via accidental correctness) without producing faithful event recovery.
• Oracle Keyframes: Providing event-centered keyframes (removing the search burden) raised accuracy to 68.6% for $N \le 5$, but performance collapsed for $N \ge 6$. This suggests that while visual access is a bottleneck, retention and accumulation of the sequence remain limiting factors even when events are perfectly localized.
• Prompting Strategies: Various prompting techniques (Direct Answer, Structured Trace, Multi-Turn, CoT, Role Prompting) yielded similarly limited gains and did not expand the reliable operating region.

Trace-Level Diagnostics

• Low Load: Models often exhibit over-reporting (VOR > 1) and accidental correctness, where the final count is right despite missing or hallucinated timestamps.
• High Load: The dominant failure mode shifts to under-reporting (VOR < 1) and omission. The model reports fewer events than occur, and the few reported events are often temporally accurate but incomplete.
• Aggregation Errors: A non-zero Reasoning Failure Rate (RFR) indicates that even when the model successfully recovers the event sequence (high trace F1), it sometimes fails to aggregate them correctly into a final count.

Natural Video Transfer

Evaluation on the TransRAC dataset confirms that the concentration of success at low event counts ($N \le 4$) persists in natural videos with camera motion and clutter, reinforcing that the failure pattern is robust across domains.

Significance and Claims

The paper argues that final-answer accuracy alone mischaracterizes temporal reasoning. A model can arrive at the correct number by guessing or aggregating hallucinated events, masking a fundamental inability to track temporal sequences.

The authors claim that:

1. Event Representation Matters: Whether an event is persistent or transient dictates initial access to evidence.
2. Visual Access is Necessary but Insufficient: Increasing frame density or providing oracle keyframes improves performance but does not eliminate the boundary imposed by sequence length and retention.
3. Staged Failure: The failure process involves distinct stages: first, accessing the event; second, retaining the sequence; and third, aggregating the count.
4. Evaluation Shift: Trace-grounded profiling shifts video evaluation from aggregate metrics to a detailed diagnostic of where temporal reasoning fails (access vs. retention vs. aggregation) and whether the reported evidence actually supports the final answer.

The study concludes that current VLMs struggle with elementary event bookkeeping as temporal load increases, a limitation that persists even with advanced prompting or increased visual sampling.