ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution

1. Problem Statement

While Multimodal Large Language Models (mLLMs) have demonstrated reasonable accuracy in answering questions about structured data (tables in Markdown, JSON, and images), a critical gap exists between answering correctly and providing evidence. In high-stakes domains (finance, healthcare, law), users require not just the answer but the specific rows and columns (attribution/citation) that support it.

The paper identifies that current mLLMs often generate correct answers through "spurious calculations" or hallucinated reasoning without grounding them in the actual data. There is a lack of comprehensive benchmarks to evaluate the fine-grained spatial attribution capabilities of mLLMs across different data representations (text vs. image) and model families.

2. Methodology

Benchmark: ViTaB-A

The authors introduce ViTaB-A, a new benchmark constructed using the HiTab dataset.

• Data: 200 tables per representation type, augmented with explicit row/column labels to enable unambiguous cell referencing (e.g., "B3").
• Representations: The benchmark evaluates models across three modalities:
  1. Markdown: Structured text.
  2. JSON: Nested structured text.
  3. Rendered Images: Visual tables (including controlled perturbations like font and color changes to test robustness).
• Task Formulation: The model is given a question ($q$), the correct answer ($a$), and the table ($T$). The task is not to generate the answer, but to localize the specific set of cells $\hat{S}$ in $T$ that support $a$. This isolates the "grounding" capability from "generation."
• Prompting Strategies: Evaluated under Zero-shot, Few-shot, and Chain-of-Thought (CoT) prompting.

Models Evaluated

The study covers a diverse range of open-source mLLM families and scales (focusing on 4B parameter models for primary analysis, with larger scales in appendices):

• Gemma-3 (4B, 12B, 27B)
• InternVL3.5 (4B, 8B, 14B, 38B)
• Qwen3-VL (2B, 4B, 8B, 32B)
• Molmo2 (4B, 8B)

Evaluation Metrics

1. Statistical Accuracy:
   • Cell-level Accuracy: Percentage of correctly cited cells.
   • Row/Column Accuracy: Separate scores for identifying the correct row vs. the correct column.
   • Precision, Recall, F1: Detailed breakdowns found in the appendix.
2. Confidence-Accuracy Alignment:
   • Internal Confidence: Derived from token-level logits (softmax probabilities) of the cited cells.
   • Verbalized Certainty: The model's explicit self-reported confidence (mapped to a 0–1 scale) via a specific "Confidence Querying Prompt" (CQP).
   • Alignment Score: Measured using the Brier Score ($1 - \text{Brier Score}$), quantifying how well the model's confidence correlates with its actual attribution accuracy.

3. Key Contributions

1. ViTaB-A Benchmark: The first exhaustive benchmark for Visual Table Attribution across text (Markdown/JSON) and image modalities, specifically designed to test spatial grounding rather than just QA.
2. Disentanglement of Capabilities: Demonstrates empirically that Question Answering (QA) accuracy and Attribution accuracy are distinct capabilities; high QA performance does not imply reliable citation.
3. Confidence Calibration Analysis: The first to benchmark open-source mLLMs on uncertainty calibration specifically for structured data attribution, revealing that high verbal confidence does not correlate with attribution correctness.
4. Modality and Granularity Insights: Provides detailed analysis on how representation (Image vs. Text) and granularity (Row vs. Column) affect model performance.

4. Key Results

A. The Attribution Gap

• QA vs. Attribution: While QA accuracy remains moderate (48–60%), attribution accuracy is significantly lower.
  • Markdown: ~25% attribution accuracy.
  • Images: ~33% attribution accuracy.
  • JSON: Near-random performance (<1% attribution accuracy), despite decent QA scores (~55%).
• Conclusion: Models can often "guess" the right answer but fail to point to the evidence.

B. Modality Effects: Images > Text

• Counter-intuitive Finding: Models perform better at attribution on images than on textual formats (Markdown/JSON).
  • Reasoning: Images preserve spatial hierarchy and visual cues (alignment, boundaries) that aid localization. Textual formats (especially JSON) require parsing nested structures without visual cues, which is a known difficulty for LLMs.
• QA Reversal: Interestingly, QA accuracy is often higher in text formats than images, suggesting models can infer answers from context without needing precise visual grounding.

C. Row vs. Column Localization

• Models are significantly better at identifying the correct Row than the correct Column.
  • Row Accuracy: ~71% (Markdown), ~61% (Images).
  • Column Accuracy: ~30% (Markdown), ~44% (Images).
• Reasoning: Rows represent complete entities (easier to match with natural language), while columns represent abstract attributes (schema linking), which is a known weak point for LLMs.

D. Confidence Misalignment

• No Reliable Indicator: There is no strong correlation between a model's confidence (internal or verbalized) and its attribution accuracy.
• Brier Scores: Alignment scores are generally low (<0.70 for most models). Even models with high internal confidence often produce incorrect citations.
• Risk: High verbal confidence in incorrect attributions creates a "confidence trap," misleading users in regulated industries.

E. Model Family Differences

• InternVL3.5 (4B) consistently outperformed other families in both QA and Attribution across modalities.
• Gemma3 and Molmo2 showed lower attribution scores, particularly in JSON.

5. Significance and Implications

• Limitations in Real-World Deployment: The findings suggest that current mLLMs are unreliable for applications requiring auditability and traceability (e.g., legal discovery, financial auditing, medical diagnosis) because they cannot reliably cite their sources.
• Training Objective Mismatch: The gap arises because standard instruction tuning (RLHF) rewards "helpful answers" but does not explicitly optimize for "faithful citation" or "spatial grounding."
• Future Directions:
  • Attribution must be treated as a first-class objective in training, not just a byproduct.
  • Benchmarks like HELM need to include structured attribution metrics.
  • Relying on confidence scores to filter hallucinations in structured data is ineffective; dedicated verification mechanisms are required.

In summary, ViTaB-A reveals that while mLLMs are becoming capable data assistants, they lack the fine-grained spatial reasoning and faithful grounding necessary to be trusted with transparent, evidence-based decision-making in structured data environments.