Technical Summary: Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

Problem Statement

While recent open-source 7–8B Video Large Language Models (Video-LLMs) have achieved reported accuracies of 37–38% on InfiniBench's "global appearance" task—which requires identifying the chronological order of a named character's outfit changes across a full TV episode—there is a critical uncertainty regarding the source of this performance. It remains unclear whether these models genuinely track character identity over time or if they rely on shallow heuristics, such as gender cues, position biases, or memorized associations, to answer correctly. The core problem is distinguishing between true character tracking (binding visual evidence to a specific named entity) and superficial pattern matching that yields high benchmark scores without genuine understanding.

Methodology

The authors propose a nine-condition diagnostic protocol to decouple benchmark accuracy from actual character-tracking capability. The evaluation focuses on the validation split of The Big Bang Theory (BBT) from InfiniBench (129 questions) and extends to Friends and Castle for cross-show analysis.

Models Evaluated:

• Open-Source (7–8B): InternVL2-8B, Qwen2.5-VL-7B, and LLaVA-NeXT-Video-7B.
• Frontier Reference: Gemini 2.5 Flash (evaluated on a partial subset due to API constraints).

Diagnostic Conditions:

1. Standard MC: Full-episode video with multiple-choice questions.
2. Text-Only: 16 black frames with the same prompt to isolate text-based guessing.
3. Targeted Frames: Replacing uniform sampling with oracle frames from ground-truth outfit-change windows to test if missing visual content is the bottleneck.
4. Name Substitution (Core Diagnostic): Swapping the named character in the question with another main-cast member while keeping the video and options identical. A model with true tracking should change its answer; a model ignoring the name should not. This is further decomposed into Same-Gender vs. Cross-Gender swaps.
5. Open-Ended (OE): Removing multiple-choice options and asking for free-text descriptions of the outfit sequence.
6. Metadata-Only: Removing the video entirely to test for pretraining contamination (memorization).
7. Frame-Count Ablation: Varying frame counts (8, 16, 32, 64) to test if increased visual input improves tracking.
8. Abstention: Adding an "I cannot determine" option to test calibration.
9. Subtitle Ablation: Adding parsed dialogue text to test if lack of textual context drives failures.

Metrics:

• Name-Substitution Sensitivity: The percentage of cases where the model changes its answer letter after a name swap.
• Collective Accuracy: The fraction of episodes where all questions are answered correctly.
• Position Bias: Chi-squared tests against uniform distribution to detect letter preference.

Key Results

1. Low Character-Tracking Sensitivity

The primary finding is that open-source models largely ignore the specific character name.

• Name Substitution: When the character name was swapped, the models changed their answer in only 4–31% of cases (depending on the model and show). This is significantly lower than the near-100% change expected if the model were tracking identity.
• Gender Bias Mechanism: The sensitivity is driven almost entirely by cross-gender swaps. When swapping a male character for a female character (or vice versa), sensitivity rises to 20–43%. However, for same-gender swaps (e.g., Sheldon to Howard), sensitivity drops to 4–17%. This indicates models use coarse gender cues but cannot distinguish between individuals of the same gender.

2. The Multiple-Choice vs. Open-Ended Gap

• Open-Source Models: When forced to generate answers without options, accuracy drops by 18–25 percentage points. Crucially, 0 out of 151 open-source open-ended responses were fully correct.
• Gemini 2.5 Flash: Shows a smaller gap (12.2 points) and achieved 14/52 fully correct open-ended answers, suggesting a more robust, though still imperfect, binding of identity.
• Implication: High MC scores for open-source models are partly an artifact of the constrained option set, which allows coarse gender filtering to eliminate wrong answers.

3. Position Bias and Calibration

• Position Bias: Open-source models exhibit strong, model-specific letter biases (e.g., LLaVA predicts 'A' 70% of the time; InternVL2 and Qwen predict 'E' ~40% of the time), which is statistically significant and unrelated to content.
• Calibration: Models rarely use the "I don't know" option (abstention rates <12%), indicating they cannot identify their own uncertainty.

4. Ruling Out Alternative Explanations

• Frame Count: Increasing frames from 16 to 32 improved overall accuracy but did not improve name-substitution sensitivity. The bottleneck is not the quantity of visual data but the binding of that data to the named entity.
• Subtitles: Adding dialogue text did not improve performance, ruling out missing textual context as the primary failure mode.
• Memorization: Metadata-only baselines scored at chance levels (~20%), ruling out pretraining memorization of show titles or canonical outfits as the driver of success.
• Annotation Structure: The study found that "difficulty" varies by show due to the interaction between episode duration and fixed frame budgets (e.g., Castle episodes are longer, causing uniform sampling to miss outfit changes more often than in BBT). This is a structural artifact, not a model capability issue.

5. Collective Accuracy

Despite aggregate scores of ~38%, 0 out of 20 BBT episodes were answered correctly in their entirety by any of the three open-source models. This confirms that correct answers are scattered guesses rather than evidence of episode-level comprehension.

Key Contributions

1. Diagnostic Protocol: A nine-condition framework that separates benchmark accuracy from character-tracking capability, specifically highlighting the role of name substitution and gender decomposition.
2. Empirical Evidence of Failure: Demonstration that current 7–8B Video-LLMs do not robustly track character identity, relying instead on coarse gender cues and position biases.
3. Toolkit Release: The authors release a diagnostic toolkit to audit character-tracking claims in future Video-LLM benchmarks.
4. Benchmark Design Implications: Recommendations to report name-substitution invariance, per-episode annotation statistics, and the MC-OE gap to prevent misleading aggregate scores.

Significance and Claims

The paper claims that the reported progress in Video-LLM benchmarks (e.g., 37–38% accuracy) is misleading regarding the specific capability of character tracking. The authors argue that these scores reflect a model's ability to filter by gender and guess based on position bias rather than the ability to "watch" and follow a specific character through a long video.

The study concludes that current improvements in Video-LLMs (scaling visual features, reasoning, and context length) are not addressing the fundamental bottleneck: binding visual identity to language tokens. The authors emphasize that without addressing this binding mechanism, Video-LLMs will continue to fail at tasks requiring genuine temporal identity tracking, regardless of aggregate benchmark scores. The work serves as a cautionary diagnostic for the field, urging a shift from aggregate accuracy metrics to identity-sensitive evaluations.