Technical Summary: TOC-Bench

Problem Statement

While Video Large Language Models (Video-LLMs) have achieved significant progress in general video understanding, their ability to maintain temporal object consistency remains insufficiently explored. Existing benchmarks primarily focus on event recognition, action understanding, or coarse temporal reasoning. However, they often rely on video-level annotations, captions, or human-written questions rather than explicit object tracks. Consequently, current evaluations may overestimate temporal reasoning capabilities by overlooking failures in object-centric temporal coherence.

Specifically, existing benchmarks rarely evaluate whether a model can consistently preserve the identity, state, and temporal continuity of the same object across:

• Occlusion and disappearance.
• Reappearance and state transitions.
• Cross-object interactions.

Furthermore, recent studies indicate that many video-language questions can be solved via "shortcuts" (e.g., relying on single frames, language priors, or unordered frame cues) rather than genuine temporal reasoning. There is a critical gap in evaluating whether models can anchor reasoning to specific object instances over time.

Methodology

TOC-Bench Construction

The authors introduce TOC-Bench, a diagnostic benchmark designed to evaluate temporal object consistency. The construction pipeline involves three stages:

1. Video Curation and Object-Track Annotation:

   • Source videos are drawn from four public datasets: Charades, Perception Test, OVIS, and MOSE.
   • Object Tracking: A VLM (Qwen3-VL-8B-Instruct) proposes distinct physical objects and people from sampled frames. These noun phrases are used as prompts for SAM3 to generate per-frame object tracks.
   • Event Derivation: Temporal events (appearance, disappearance, occlusion, reappearance, interaction) are derived from the retained object tracks.
   • Quality Control: Low-quality, fragmented, or ambiguous tracks are removed. A post-processing step repairs ID splits and flags ID swaps to ensure track stability.

2. QA Construction and Generation:

   • Reasoning Units: Structured reasoning units are created from object tracks and event timelines, specifying the queried object, event, diagnostic dimension, and correct answer. This step generates 547,610 reasoning units.
   • Skeleton Synthesis: Reasoning units are converted into question skeletons (fixing structure, options, and labels) covering 10 diagnostic dimensions and 4 task formats (multiple-choice, statement-pair, numerical counting, event ordering).
   • Surface Realization: An LLM (GPT-5.4-mini) polishes the wording of the skeletons without altering the deterministic answer labels or structured metadata.
   • Hallucination Injection: Variants are created where the subject or event does not exist in the video to test for hallucination robustness.

3. Filtering and Verification:

   • Three-Layer Temporal-Necessity Filter: To eliminate shortcut-solvable questions, a three-layer filter is applied:
     1. Text-only filter: Removes items answerable from text alone.
     2. Single-frame filter: Removes items answerable from individual frames.
     3. Frame-shuffled filter: Removes items where the answer does not depend on the original temporal order.
   • This process reduces the candidate pool from 45,527 to 17,900 temporally dependent items.
   • Human Verification: 3,000 filtered items are manually verified for clarity, evidence validity, and label correctness, resulting in the final 2,323 high-quality QA pairs across 1,951 videos.

Evaluation Protocol

• Models Evaluated: 23 representative Video-LLMs, including closed-source systems (e.g., GPT-5.5, Kimi-K2.6), open-source standard models (e.g., Qwen2.5-VL, InternVL3), and reasoning-style variants.
• Input Protocol: Models are evaluated under a controlled frame-based protocol (typically 32 uniformly sampled frames) to ensure reproducibility and isolate visual evidence budgets.
• Metrics: Overall accuracy, per-dimension accuracy, and Hallucination Diagnostic Accuracy (HDA) for variants involving nonexistent subjects or events.

Key Contributions

1. Definition of Temporal Object Consistency: The paper establishes temporal object consistency as a dedicated evaluation capability distinct from general event-level or video-level reasoning. It focuses on maintaining object identity, state, and persistence across time.
2. TOC-Bench Dataset: An object-track grounded diagnostic benchmark containing 2,323 human-verified QA pairs. Unlike previous benchmarks, every queried subject is anchored to a per-frame object track, and answers are derived from structured object-event timelines.
3. Diagnostic Protocol:
   • Temporal-Necessity Filtering: A rigorous protocol that removes 60.7% of candidate QA pairs to ensure questions require temporally ordered visual evidence.
   • Hallucination Awareness: Integration of variants testing whether models can reject plausible but visually unsupported objects or events.
4. Comprehensive Evaluation: The benchmark covers 10 diagnostic dimensions across three difficulty tiers (single-object, cross-time consistency, cross-object consistency) and supports four deterministic task formats.

Experimental Results

Experiments on 23 Video-LLMs reveal that temporal object consistency remains a major unsolved challenge:

• Human-Model Gap: Human annotators achieve 89.3% overall accuracy, while the best-performing model (GPT-5.5) reaches only 47.1%.
• Specific Weaknesses: Models exhibit substantial weaknesses in:
  • Event Counting: (e.g., GPT-5.5 scores 18.3% on event counting vs. 68.9% on event existence).
  • Event Ordering: (e.g., 24.8% accuracy).
  • Identity-Sensitive Reasoning: Struggles with maintaining identity across occlusion and reappearance.
  • Hallucination Verification: Models often over-assume the existence of objects or events not present in the video.
• Model Trends:
  • Closed-source models generally outperform open-source models, but the gap to human performance remains large.
  • "Thinking" or reasoning-style variants (e.g., Qwen3-VL-8B-Thinking) show selective improvements but do not consistently outperform standard models, suggesting that longer reasoning traces cannot compensate for missing object-level temporal evidence in the visual representation.
  • Performance varies significantly across dimensions; statement-pair questions are generally easier than numerical counting or event ordering.

Significance and Claims

The paper claims that temporal object consistency is a major bottleneck for current Video-LLMs. The findings suggest that strong performance on general video understanding benchmarks does not imply reliable object-level temporal reasoning.

• Diagnostic Value: TOC-Bench provides a focused platform for diagnosing specific failures in object-aware temporal reasoning, moving beyond broad average-case evaluations.
• Design Implications: The failure patterns suggest that current mechanisms (sparse frame sampling, visual-token compression, global temporal aggregation) may discard the frame-level continuity required for counting repeated events or distinguishing occlusion from leaving the frame.
• Future Directions: The authors suggest that future models may benefit from object-centric memory, tracking-aware visual tokens, and training objectives that explicitly model object identity, event timestamps, and cross-object temporal relations.

The resource is made available to facilitate further research in this area.