Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MME

Technical Summary: "Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MME"

Problem Statement
The paper addresses a prevailing assumption in the adaptation of Vision-Language Models (VLMs) to video question answering (Video-QA): that forcing models to generate Chain-of-Thought (CoT) reasoning steps before answering improves reliability and accuracy. While CoT is widely adopted, its functional utility in video contexts remains unverified. The authors challenge the "boilerplate-chain" null hypothesis—the prediction that forced CoT chains might be largely input-invariant (i.e., the reasoning text remains similar regardless of the video input) yet still shift the final answer due to format effects, without the model meaningfully "reading" the visual input. Conversely, they also question whether even if chains are video-conditioned, this necessarily translates to improved Multiple-Choice Question (MCQ) accuracy.

Methodology: A Three-Probe Evaluation Recipe
The authors propose a specific, three-probe evaluation recipe instantiated on the Video-MME benchmark using the Qwen2.5-VL family (7B and 32B parameter variants). The pipeline is designed to decouple chain generation quality from final answer utility.

1. Probe 1: Paired Accuracy with Format Controls

   • Design: Evaluates performance under four conditions: Direct (answer only), CoT (forced reasoning steps), Answer-First (answer then reasoning), and No-Video (text-only).
   • Scoring: Uses two regex-based scorers to extract answers:
     • Strict: Requires explicit tags (e.g., "Answer: X") or specific line-initial formatting.
     • Permissive: Includes a fallback rule to coerce refusal-style outputs into a letter choice.
   • Statistics: Reports paired McNemar tests, 50k bootstrap confidence intervals (CIs), and Holm-corrected $p$-values over a manuscript-declared primary family of four tests.

2. Probe 2: Counterfactual Video-Swap Diagnostic

   • Design: Re-evaluates CoT responses where the original video is swapped with a different video of the same task type.
   • Metrics:
     • Chain Token-Jaccard: Measures overlap between the reasoning chain of the real video and the swapped video.
     • Final Letter Flip Rate: Measures how often the predicted answer changes when the video changes.
     • Original-Label Retention: Checks if the model retains the answer for the original video when shown the swapped video (a diagnostic for refusal vs. coincidence).
   • Goal: To test if chains are truly input-conditioned. A "boilerplate" chain would show high Jaccard similarity and low flip rates; an input-conditioned chain would show the opposite.

3. Probe 3: Visual-Degradation Ladder

   • Design: Evaluates CoT performance across a monotonically decreasing visual signal ladder: Real $\to$ Shuffled Frames $\to$ Single Frame (repeated) $\to$ Black Frames. Two contamination conditions (swapped task/domain) are reported separately.
   • Metrics: Spearman correlation ($\rho$) between visual degradation and accuracy/chain similarity.
   • Goal: To determine if the reasoning chain utilizes visual signal and if accuracy degrades in tandem with visual information loss.

Key Results
The evaluation on Qwen2.5-VL (7B and 32B) yields a two-part finding:

1. Chains are Strongly Video-Conditioned:

   • The "boilerplate-chain" null is rejected. When the video is swapped, chain token-Jaccard drops significantly (to 0.14 for 32B and ~0.12 for 7B), and final letter flip rates are high (67% for 32B, ~60% for 7B).
   • Qualitative examples show the model explicitly describing the content of the swapped video and refusing to answer if the content does not match the options, rather than generating a generic template.

2. Video-Conditioned Chains Do Not Improve Accuracy (and May Harm):

   • 32B Model: Forced CoT does not improve MCQ accuracy over direct answering. Under strict scoring, CoT shows a slight drop (-5.33 pp, $p=0.048$) that does not survive Holm correction. Under permissive scoring, the drop is smaller (-2.33 pp) and non-significant. A sign flip occurs in parsed-only subsets, attributed to parse-rate selection effects rather than genuine improvement.
   • 7B Model: Forced CoT produces a statistically supported drop in accuracy. Under the strict scorer, the drop is -7.32 percentage points ($p=0.003$, Holm-adjusted $p=0.012$). This is the only result in the primary family that survives Holm correction at $\alpha=0.05$. The authors hypothesize this is due to error propagation, where the smaller model commits to intermediate inferences that degrade the final answer.
   • Visual Ladder: Accuracy and chain similarity both decrease monotonically as visual signal degrades (Real $\to$ Black), confirming the chains use visual input, but this grounding does not correlate with accuracy gains in the CoT format.

Significance and Claims
The paper does not claim that CoT is universally useless, nor does it claim these results generalize beyond the Qwen2.5-VL / Video-MME instantiation. Instead, its significance lies in the evaluation methodology and the specific discovery of a decoupling between chain grounding and task utility.

• Methodological Contribution: The authors provide a "three-probe recipe" to rigorously test CoT in multimodal settings. They argue that standard accuracy comparisons are insufficient; evaluators must separately measure input-conditioning (via swap probes) and parse-rate sensitivity (via strict vs. permissive scoring).
• Empirical Finding: The paper demonstrates that a model can generate reasoning traces that are highly sensitive to visual input (proving it "sees" the video) yet fail to leverage that reasoning to improve final answers, and in the case of smaller models, actively degrade performance.
• Evaluation Lesson: The authors caution that "grounded reasoning traces" are not a sufficient condition for downstream gains. They advocate for reporting parse rates alongside accuracy, using input-substitution probes to verify chain conditioning, and treating "CoT helps/doesn't help" as a multi-faceted question requiring specific diagnostic tools rather than a single aggregate score.

The authors explicitly state that their custom regex scorer is not the official Video-MME evaluator and that their primary significance choice (strict scoring) was post-hoc. They release raw responses and a recomputation script to ensure all numbers are reproducible, inviting future work to validate these findings on the full benchmark and other model families.