Technical Summary: Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

1. Problem Statement

Current evaluation practices for Video Large Language Models (Video LLMs) rely heavily on aggregate scaling curves, which plot average performance against increasing visual budgets (e.g., frame count or resolution). These curves typically appear smooth or saturating, leading to the interpretation that increasing visual input is either beneficial or merely redundant.

The authors argue that this aggregate view conceals significant item-level instability. At any given budget transition, a model may flip from incorrect to correct on some items while simultaneously flipping from correct to incorrect on others. These opposing flows cancel out in the mean, creating a "stable" aggregate curve that masks substantial item-level churn. This phenomenon, termed visual confusion, suggests that no single visual budget is optimal for all items, and that higher budgets can actively harm performance on specific inputs.

2. Methodology

The paper introduces a paired scaling-trajectory audit to expose these hidden dynamics.

• Paired Item-Level Trajectories: Instead of reporting a single score per budget, the authors track the response trajectory of a frozen model on the same item across a controlled grid of visual configurations (varying frame count, resolution, sampling policy, and temporal-spatial allocation).
• Matched-Grid Metrics: To ensure fair cross-model comparison, the study utilizes matched grids (e.g., {text, 16f, 32f, 64f}) that respect the architectural context ceilings of different models.
• Key Metrics:
  • Oracle Headroom: The difference between the best fixed accuracy (using a single budget for all items) and the item-level oracle accuracy (the maximum accuracy achievable if the optimal budget were selected for each item individually).
  • Visual Confusion: The rate of items that are correct at a lower-cost configuration but incorrect at a higher-cost one.
  • Text Overwrite: A specific subset of confusion where an item correct in text-only mode becomes incorrect when visual inputs are added.
  • Churn: The sum of items flipping from wrong-to-correct and correct-to-wrong, distinct from the net aggregate change.
• Experimental Scope: The study covers five frozen open Video LLMs (Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL3-8B, InternVL3.5-8B, LLaVA-NeXT-Video-7B) across three architecture families. Evaluations span four multiple-choice benchmark splits (Video-MME v1/v2, MLVU), open-ended QA/summarization (MLVU), and controlled dialogue generation (AVSD).
• Interventions: The authors test robustness across raw-video vs. cached pipelines, various sampling policies (uniform, random, dense-start), and iso-compute allocations (trade-offs between frame count and spatial resolution).

3. Key Results

Aggregate Stability vs. Item Instability

• Cancellation Effect: On the Video-MME v1 short split, increasing frames from 128 to 256 resulted in a net accuracy change of only -0.7 points. However, this masked 23 items becoming wrong and 18 becoming correct, representing a 5.2% state change in the benchmark population.
• Oracle Headroom: Across four models and four MCQA splits, the item-level oracle headroom ranges from 8.8 to 18.9 accuracy points. This indicates a significant gap between current fixed-budget deployment and the theoretical maximum performance achievable with adaptive budgeting.
• Visual Confusion Rates: Between 12.5% and 25.5% of items exhibit visual confusion (correct at lower budget, wrong at higher budget).
• Text Overwrite: A non-negligible fraction of items (5.0–7.3% of total, or 13.2–17.9% of text-correct items) are correctly answered via text alone but become incorrect when visual inputs are introduced. This effect is particularly strong in "Counting Problems."

Generalization Across Tasks and Protocols

• Open-Ended Generation: The phenomenon persists beyond multiple-choice. On MLVU generation tasks, Token-F1 oracle gaps range from 2.7 to 3.7 points. On AVSD dialogue generation, gaps range from 3.8 to 4.8 points. Even when the mean quality improves with budget, a significant fraction of items suffer score regressions.
• Robustness: The heterogeneity persists across different resolutions, sampling policies, and execution pipelines (raw vs. cached). While the exact set of confused items varies slightly by protocol, the population-level structure of cancellation and complementarity remains consistent.
• Iso-Compute Heterogeneity: When holding total pixel count constant (e.g., ~9.7M pixels), different temporal-spatial allocations (e.g., 32 frames at high resolution vs. 128 frames at low resolution) are optimal for different subsets of items. No single allocation is universally best.

Operational Implications

• Confidence Cascade: The authors demonstrate an operational use case: a "confidence cascade" that starts with a low budget (e.g., 16 frames) and escalates only if the model's confidence is low. This approach matches the accuracy of a fixed 128-frame budget while reducing the average shared frame cost by 31.7% and wall-clock latency by 28.2% (with frame sharing).
• Sampling Sensitivity: A controlled sampling intervention recovered 29.0% of terminal regressions (items that were wrong at a high budget but could be made correct by changing the sampling policy), suggesting that evidence composition is a controllable factor in these failures.

4. Contributions

1. Evaluation Protocol: Formulation of Video LLM scaling as paired item-level response trajectories, providing matched-grid metrics that expose bidirectional cancellation hidden by aggregate curves.
2. Empirical Coverage: Establishment of broad empirical evidence across five models, three architecture families, and diverse task formats (MCQA, generation, dialogue), demonstrating that item-level complementarity is a consistent population-level phenomenon.
3. Protocol Independence: Separation of robust population-level structure from protocol-sensitive item labels through full sampling, resolution, and raw/cache analyses.
4. Auditing Artifact: Release of a comprehensive dataset containing ~0.13M per-item model outputs, derived annotations (trajectories, confusion tags), protocol provenance, and analysis code.

5. Significance and Claims

The paper claims that the standard view of Video LLM scaling is incomplete. The "smooth" aggregate curves often cited in literature are a statistical artifact of opposing item-level transitions.

• Measurement Shift: The authors argue for a shift from reporting only aggregate means to reporting rescued and harmed flows alongside the mean. This is essential for interpreting near-zero net changes.
• Adaptive Inference: The existence of significant oracle headroom and item-specific optima suggests that adaptive inference (allocating compute dynamically per item) is a viable and necessary strategy for efficient Video LLM deployment, rather than simply increasing the budget for all inputs.
• Benchmark Design: The findings highlight the need for matched-grid comparisons to isolate behavioral differences from architectural context ceilings and suggest that benchmark designs should account for text-overwrite shortcuts and letter-prior biases.

The work does not claim to solve the underlying causes of visual confusion (which remain heterogeneous and not fully explained by simple low-level statistics) but provides the necessary measurement framework and data to audit these behaviors and develop targeted mitigation strategies.