Technical Summary: TABVERSE

Problem Statement

While Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, the specific role of table representation remains under-explored. In real-world scenarios, identical table content can appear in diverse structural formats (e.g., HTML, Markdown, LaTeX) or as rendered images. Existing benchmarks often allow table content, format, layout, and modality to vary simultaneously, making it difficult to isolate the effects of representation alone. Consequently, it is unclear whether performance differences stem from the model's reasoning capabilities or its sensitivity to specific serialization formats and input modalities.

Methodology

The authors introduce TABVERSE, a controlled multimodal benchmark designed to disentangle representation effects from content variation.

Dataset Construction

TABVERSE is constructed from held-out evaluation splits of five established TableQA datasets: FEVEROUS, HYBRIDQA, TABFACT, SQA, and WikiTableQuestions.

• Alignment: The benchmark aligns identical table content across three structural text formats (HTML, LaTeX, Markdown) and their corresponding rendered images.
• Normalization: Conversion utilities were adapted to handle dataset artifacts (e.g., missing values, special characters) and ensure consistent row/column ordering. Rendered images were generated under standardized layouts (font size, padding, width) to ensure visual alignment across formats.
• Tagging: Each question is tagged with a category (Simple Lookup, Conditional Lookup, Multi-item Lookup, Aggregation/Arithmetic, Comparison/Extremum, Single-step/Multi-hop Binary Verification) and a difficulty label (Easy/Hard). Difficulty is estimated via zero-shot QA performance on rendered images.
• Evaluation Set: A balanced set of 700 question–table pairs covering 629 unique tables was selected, ensuring equal distribution across difficulty levels and question categories.

Tasks

TABVERSE evaluates models on three complementary tasks:

1. Question Answering (QA): Predicting an answer given a question and a table in structured text or rendered image form.
2. Structural Understanding Capability (SUC): Probing structure understanding through boundary detection, size estimation, coordinate lookup, and index-based retrieval (row/column).
3. Structure Reconstruction (SR): Reconstructing a table from a rendered image into a target text format (HTML, Markdown, or LaTeX).

Experimental Setup

The study evaluates a diverse range of LLMs and VLMs (including general-purpose and table-specialized models like Qwen, InternVL, Gemma, LLaVA, and proprietary models like GPT-5.2 and Gemini-3) using three pipelines:

• VLM-Image: Models receive rendered table images.
• VLM-Text: Models receive structured text representations (isolating modality effects within VLMs).
• LLM-Text: Text-only models receive structured text.

Evaluation metrics include Exact-Match (EM) accuracy for QA and SUC, and GriTS (Grid Table Similarity) for SR, alongside measures of syntactic usability (e.g., LaTeX compilation success).

Key Contributions

1. Controlled Evaluation Framework: The paper formulates cross-format and cross-modality table understanding as a controlled problem where table content is fixed while structural format and input modality vary.
2. TABVERSE Benchmark: The introduction of an aligned multimodal benchmark featuring HTML, LaTeX, and Markdown representations, corresponding rendered images, and a 700-sample balanced evaluation set with category and difficulty tags.
3. Comprehensive Benchmarking: The authors evaluate LLMs and VLMs across matched text-only and image-based inputs, revealing how format and modality choices alter model behavior. They specifically analyze how SR errors separate into reconstruction quality and output usability.

Results

Question Answering (QA)

• Representation Impact: Structured text often outperforms rendered images, but the magnitude of this gap depends on the model and task. For instance, Gemma-3 benefits significantly from text inputs, whereas SmolVLM2 and InternVL3.5 perform similarly or better on images.
• Format Sensitivity: In text pipelines, symbolic format matters significantly (HTML often being the most robust), whereas rendered-image performance is generally consistent across HTML, LaTeX, and Markdown sources.
• Task Difficulty: Verification questions are generally easier, while multi-item lookup and aggregation tasks remain challenging. Performance drops substantially from Easy to Hard questions.

Structural Understanding Capability (SUC)

• Modality and Format: SUC is generally stronger with structured text than with rendered images, particularly for tasks involving row boundaries and header handling. However, text input does not fully resolve bottlenecks; row retrieval and cell lookup remain difficult across all pipelines.
• Indexing Conventions: Models are highly sensitive to indexing conventions. Explicit prompts clarifying header exclusion and 0-based indexing significantly improve performance on first-cell detection and table partitioning, suggesting many errors stem from models treating headers as the first data row.
• Partial Recovery: Field Accuracy metrics reveal that low Exact-Match scores often reflect incomplete or shifted localization rather than total failure to understand the structure.

Structure Reconstruction (SR)

• Topology vs. Content: Models recover table topology (layout) more reliably than exact cell text. GriTS-Topology scores are consistently higher than GriTS-Content scores.
• Usability Challenges: High reconstruction scores do not guarantee syntactic usability. While HTML and Markdown outputs are often valid, LaTeX reconstruction remains a significant bottleneck, with many models producing non-compilable code even when the structure is partially correct.
• Format Conversion: Cross-format conversion (e.g., Image to LaTeX) introduces little additional difficulty compared to same-format reconstruction for strong models, suggesting errors arise primarily from table understanding and target-format generation rather than the conversion process itself.

Significance and Claims

The paper claims that table representation is a key factor in reliable table evaluation. The findings demonstrate that:

• Scaling alone does not guarantee better table understanding; model behavior varies significantly based on input modality and serialization format.
• HTML is often the most robust text format for text-input pipelines, while LaTeX reconstruction remains a persistent challenge due to syntactic constraints.
• Row-sensitive tasks and syntactically usable LaTeX generation are current failure points for even state-of-the-art models.

The authors conclude that future work should extend TABVERSE to more realistic settings (e.g., noisy PDFs, multi-table documents) and explore methods for improving cross-format consistency and representation-aware decoding. The benchmark is intended to support transparent and reproducible evaluation of LLMs and VLMs in academic research.