💬 NLP

The Mighty ToRR: A Benchmark for Table Reasoning and Robustness

This paper introduces ToRR, a comprehensive benchmark designed to evaluate the reasoning capabilities and robustness of AI models across diverse table formats and domains, revealing that even strong models exhibit brittle behavior and highlighting the critical importance of testing multiple formats and prompts for reliable performance estimation.

Original authors: Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer

Published 2026-02-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to manage your company's spreadsheets. You have a stack of resumes (the AI models), and you want to know who is the best at reading tables, doing math, and finding facts.

This paper, titled "The Mighty ToRR," is like a massive, rigorous job interview designed to test these AI assistants. The researchers from IBM, Stanford, and MIT realized that while we know AI is good at writing poems or chatting, we don't really know if they can actually handle the boring, messy, real-world data we use every day.

Here is the breakdown of their findings, using some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Trap

Imagine you ask your assistant, "How much did we spend on coffee last month?"

Scenario A: You show them a neat, printed Excel sheet.
Scenario B: You hand them a crumpled receipt where the numbers are written in a different order.
Scenario C: You type the data into a chat box as a simple list.

You would expect the assistant to give you the same answer in all three cases, right? The paper reveals that current AI models are surprisingly fragile. They are like a student who memorized the answer key for a specific test format but fails if you change the font or shuffle the questions.

2. The Solution: The "ToRR" Benchmark

The researchers built ToRR (Table Reasoning and Robustness). Think of this as a "stress test" gym for AI.

The Workout: They took 10 different types of table tasks (like financial math, scientific data, and Wikipedia facts).
The Obstacles: For every single question, they didn't just ask it once. They asked it 35 different ways.
- They changed the format (like turning a table into a CSV file, HTML code, or a Markdown list).
- They shuffled the deck (randomizing the order of rows or columns, or even flipping the table sideways).

If the AI is truly smart, it should get the answer right no matter how the data is presented. If it gets confused just because the rows were shuffled, it's not "robust."

3. The Shocking Results: "Brittle" Brains

The results were a bit of a wake-up call for the AI community:

The "Brittle" Effect: Even the most advanced, expensive AI models (like GPT-4o or Claude 3.5) struggled. They were "brittle," meaning they broke easily under small changes. One moment they were a genius accountant; the next, with the rows shuffled, they were guessing.
No Magic Format: The researchers hoped to find that "Markdown" or "JSON" was the perfect way to talk to AI. They were wrong. No single format worked best for everyone. Sometimes a model liked CSV; other times it preferred HTML. It's like trying to find the perfect language to speak to a person who changes their mind every 5 minutes.
Size Doesn't Always Matter: Bigger models (with more "brain power") were generally better, but the gap between the "smartest" and the "okay" models wasn't huge. They all had the same fundamental weakness: they couldn't handle the messiness of real data consistently.

4. The Big Lesson: Don't Trust a Single Test

This is the most important takeaway for anyone using AI.

The "One-Shot" Lie: Most benchmarks ask a model a question once and declare a winner. The paper argues this is like judging a chef by letting them cook one dish once. If they had a bad day or the ingredients were slightly different, you'd get a wrong rating.
The "Many-Shot" Truth: The researchers found that if you ask the model the same question in multiple ways (different prompts, different formats), you get a much more reliable picture of its true intelligence.
The Magic Trick: They discovered that asking the same question 10 different ways is just as effective as testing the model on 100 different questions. It's a cheat code for reliability: instead of needing a massive library of test questions, you just need to ask the same questions in different voices.

Summary

The paper tells us that while AI is getting smarter, it is still unreliable when dealing with tables. It's like a brilliant student who panics if the teacher changes the font on the exam paper.

ToRR is a new tool that forces AI to prove it understands the meaning of the data, not just the look of the data. The authors suggest that to truly trust an AI in the real world, we must stop testing it with a single, perfect prompt and start testing it with a chaotic variety of formats to see if it can truly reason through the noise.

1. Problem Statement

Despite the ubiquity of tabular data in real-world applications (finance, science, news), Large Language Models' (LLMs) ability to reason over tables remains under-explored and poorly understood. Existing benchmarks often suffer from three critical limitations:

Narrow Scope: They typically focus on specific tasks (e.g., only Question Answering) or narrow domains, failing to capture the full spectrum of table reasoning skills.
Lack of Robustness Assessment: They evaluate models on a single, fixed table format (e.g., Markdown or JSON), ignoring the fact that real-world tables appear in diverse, semantically equivalent formats.
Evaluation Fragility: They do not account for how minor variations in prompt structure (serialization or perturbation) drastically alter model performance, leading to unreliable rankings.

The authors argue that without assessing robustness across formats, current evaluations cannot reliably determine which models are truly capable of table reasoning.

2. Methodology: The ToRR Benchmark

To address these gaps, the authors introduce ToRR (Table Reasoning and Robustness), a comprehensive benchmark designed to measure both performance and consistency.

A. Dataset Construction

ToRR aggregates 10 datasets covering 6 distinct task types across diverse domains (Finance, Wikipedia, Science, etc.):

Table QA: FinQA, WikiTableQuestions (WikiTQ), TableBench (Data Analysis, Fact Verification, Numerical Reasoning).
Fact Verification: TabFact.
Table-to-Text Generation: QTSumm, SciGen, NumericNLG.
Classification: TURL Column Type Annotation (CTA).

The datasets were manually analyzed to identify three core reasoning skills required:

Knowledge Extraction: Retrieving specific fields/entities.
Textual Reasoning: Combining text and data to deduce conclusions.
Numerical Reasoning: Performing calculations and aggregations.

B. Prompt Configurations & Robustness Testing

The core innovation of ToRR is the systematic variation of input prompts. For every test example, the table is presented in 35 distinct prompt configurations:

7 Serializations: Different text formats representing the same table (HTML, CSV, JSON, Markdown, Indexed Row Major, DataFrame, Concatenation).
4 Structural Perturbations: Changes to the table structure that preserve semantics but alter the layout (Row Shuffling, Column Shuffling, Transposition, Adding Empty Rows).
Control: 7 configurations without perturbations.

This design tests whether a model's performance is consistent regardless of how the data is presented.

C. Metrics

Performance ( $P$ ): The average score across all prompt configurations and examples.
Robustness ( $R$ ): Defined as $1 - \text{Average Range}$ . It measures the variance in a model's score across the 35 configurations for the same example. A high $R$ indicates the model is insensitive to format changes; a low $R$ indicates brittle behavior.

D. Experimental Setup

Models: 14 leading LLMs (7 families), including closed models (GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro) and open models (Llama-3.1, Qwen2, Mixtral).
Protocol: 100 examples per dataset, 5-shot prompting, greedy decoding.
Total Scale: ~110–140 million tokens per model.

3. Key Contributions

Comprehensive Benchmark: ToRR is the first benchmark to systematically evaluate table reasoning across diverse tasks, domains, and multiple input formats.
Robustness Measurement: It introduces a formal metric for robustness, revealing that current models are highly sensitive to prompt variations.
Meta-Evaluation Insight: The paper demonstrates that evaluating across multiple prompt configurations is more effective for reliable ranking than simply increasing the number of test examples.
Open Resources: The authors release the leaderboard, full inference outputs, and code (via the Unitxt library).

4. Key Results & Analysis

A. Model Capabilities are Limited

Low Absolute Performance: Even the best models (e.g., Claude-3.5-Sonnet, GPT-4o) achieve only moderate scores (max ~0.50 on a 0-1 scale).
Narrow Gaps: The performance difference between top-tier and mid-tier models is often small and statistically insignificant in practical terms (small Cohen's $d$ ).
Closed vs. Open: Closed proprietary models generally outperform open-weight models, but the gap is not absolute across all tasks.

B. Extreme Brittleness (The "Striking Pattern")

Inconsistency: Models exhibit wildly different scores for the same question depending on the table format.
- Example: A model might score 0.9 on a Markdown table but 0.1 on a shuffled CSV for the same example.
No "Best" Format: No single serialization (e.g., JSON vs. HTML) consistently yields superior performance across all models or tasks.
Perturbation Sensitivity: Structural changes like row shuffling or transposing cause significant performance drops, indicating models lack a stable, generalized internal representation of tables.

C. The Value of Multiple Prompts

Reliability Boost: The paper proves that testing a model with multiple prompt configurations significantly increases the reliability of the evaluation (measured by Kendall's $W$ agreement on rankings).
Substitution Effect: Adding prompt variations can compensate for a smaller test set. For instance, 50 examples with 2 prompts yield similar ranking reliability to 100 examples with 1 prompt.
Conclusion: A single-prompt evaluation is unreliable; robust assessment requires multi-format testing.

5. Significance and Implications

Rethinking Evaluation: The paper challenges the standard practice of using a single prompt format for LLM benchmarks. It argues that current leaderboards may be misleading due to "format luck" rather than true reasoning capability.
Real-World Relevance: Since real-world data comes in messy, varied formats, models that fail ToRR's robustness test are likely to fail in production environments where data formatting is not controlled.
Future Directions: The findings suggest that improving LLMs requires not just better reasoning, but better generalization across input modalities. The authors advocate for adopting multi-prompt evaluation protocols in future benchmarking to ensure reliable AI development.

In summary, ToRR reveals that while LLMs show promise in table reasoning, they remain brittle and inconsistent. The benchmark provides a necessary tool to move beyond simple performance rankings toward a more nuanced understanding of model reliability in structured data tasks.