Technical Summary: Encoded but Not Routed

Problem Statement

Multimodal Large Language Models (VLMs) are increasingly deployed for scientific claim verification, a task requiring the determination of whether a claim is supported or refuted by evidence in research papers. Prior work has established a significant performance gap: models perform substantially better when evidence is presented as a table compared to when it is presented as a chart, even when the underlying data is identical.

The central question addressed by this paper is whether this "table-chart gap" stems from a perceptual failure (the model fails to extract or encode visual information from charts) or a utilization failure (the model encodes the information but fails to route it effectively to the prediction stage). Distinguishing between these is critical, as they require fundamentally different interventions (e.g., better vision encoders vs. architectural changes to information routing).

Methodology

The authors employ a diagnostic approach using layer-wise linear probing and attention analysis on three open-weight VLMs (Qwen2.5-VL-7B, Qwen2.5-VL-32B, and InternVL3-8B) from two distinct model families.

1. Dataset: The study utilizes SciTabAlign+, a content-controlled benchmark pairing 162 unique scientific claims with semantically equivalent evidence in two formats: structured JSON tables and four chart variants (basic bar, symbol bar, line, and swapped charts).
2. Linear Probing:
   • Objective: To determine if task-relevant signals (Supported/Refuted) are decodable from intermediate hidden states.
   • Settings: Probes are trained at every layer $l$ using two aggregation strategies:
     • Last-token: Using the hidden state of the final input token (the prediction position).
     • Mean-pool: Averaging hidden states across all input tokens to capture distributed information.
   • Control: Probes are also trained on claim text alone to rule out text leakage.
   • Metric: Direction-corrected Area Under the Receiver Operating Characteristic Curve (AUROC).
3. Attention Analysis:
   • The authors compute the fraction of attention the last token directs toward image tokens, normalized by the proportional baseline (the ratio of image tokens to total tokens).
   • This measures whether the model attends to chart evidence in proportion to its presence in the input.
4. Chain-of-Thought (CoT) Ablation:
   • Models are prompted to verbally describe the chart before making a prediction to test if explicit verbalization bridges the gap.

Key Results

1. Chart Information is Encoded but Not Routed

The primary finding is that chart-relevant signals are recoverable from intermediate representations but fail to reach the prediction position.

• Mean-pool vs. Last-token: Mean-pooled probes achieve high AUROC (84–89%) on chart variants across all models, significantly outperforming last-token probes (64–73%). In contrast, table evidence shows the opposite trend, with last-token probes performing better.
• Dissociation: A linear classifier trained on mean-pooled representations often outperforms the model's own inference accuracy on chart variants (e.g., InternVL3-8B: 79.0% probe accuracy vs. 56.2% model accuracy). This confirms that the information is present in the model's internal states but is not effectively utilized for the final decision.

2. Distinct Failure Modes Across Model Families

Attention analysis reveals that the "routing failure" manifests differently depending on the model architecture:

• Qwen Family (Routing Failure): These models exhibit a severe attention routing failure. At the final layer, they attend to image tokens at only 4–11% of the proportional baseline. The chart evidence is largely bypassed when forming the prediction.
• InternVL3-8B (Integration Failure): This model maintains near-proportional attention to image tokens (93% of baseline) throughout most layers. However, it still fails to outperform its mean-pool probe. This suggests a post-attention integration failure: the model attends to the chart but fails to integrate that attended information into the final prediction logic.

3. Impact of Chain-of-Thought

The CoT ablation further distinguishes the failure modes:

• Qwen Models: CoT prompting degrades performance (macro-F1 drops by 3.3–13.5%). Since these models do not attend to the chart, forcing them to verbalize content leads to hallucinations and unreliable descriptions.
• InternVL3-8B: CoT prompting improves performance (macro-F1 increases by 4.6). This indicates the model can access and describe chart content but does not spontaneously use it for prediction without explicit instruction.

Significance and Contributions

The paper reframes the table-chart gap not as a failure of encoding (perception) but as a failure of routing (utilization).

• Diagnostic Insight: The study demonstrates that output evaluation alone is insufficient to diagnose format sensitivity. Internal analysis (probing and attention) is required to distinguish between models that "don't see" the chart and models that "see but ignore" it.
• Architectural Specificity: The findings highlight that different model families suffer from distinct internal mechanisms of failure. A single intervention (e.g., better vision encoders) is unlikely to solve the problem for all models; instead, solutions must address specific routing or integration bottlenecks.
• Implications for Verification: For high-stakes scientific workflows, the presence of encoded information does not guarantee its use in the final decision. Researchers must exercise caution when deploying VLMs for chart-based verification, as the models may possess the necessary information internally without reflecting it in their outputs.

The authors conclude that the gap is a "failure of how encoded visual information is routed at prediction time," a finding that necessitates examining internal computation rather than relying solely on task performance metrics.