MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark

This paper introduces MMTU, a large-scale benchmark comprising over 28,000 questions across 25 real-world expert-level table tasks, designed to comprehensively evaluate and reveal the significant limitations of current frontier models in understanding, reasoning, and manipulating structured tabular data.

The Gist

Imagine you have a super-smart assistant (an AI) that can read books, write essays, and even solve math problems. You might think, "Great! It can handle my spreadsheets too, right?"

Not quite. That's the problem this paper tackles.

The Problem: The "Excel" Gap

Think of Large Language Models (LLMs) like a brilliant librarian who has read every book in the world. They are amazing at understanding text. But tables (like Excel sheets, databases, or financial reports) aren't just text; they are structured grids with rows and columns that hold specific relationships.

Until now, we've only tested these AI librarians on simple "table questions" like, "What is the total sales for July?" (This is like asking the librarian to find a specific sentence in a book).

But real-world experts—data analysts, accountants, and engineers—don't just read tables; they manipulate them. They need to:

• Fix broken data (like finding a missing piece in a puzzle).
• Combine two different spreadsheets (like merging two guest lists for a party).
• Write code to transform messy data into a clean report.
• Spot errors that don't make sense (like a person's age listed as "200").

Current AI models are terrible at these "expert" tasks. They get lost in the grid, mix up rows and columns, or hallucinate facts.

The Solution: MMTU (The "Table Olympics")

The authors created a new benchmark called MMTU (Massive Multi-Task Table Understanding). Think of this as the Olympics for AI on Tables.

Instead of just asking simple questions, MMTU throws 28,000 different "events" at the AI. These events are drawn from decades of real computer science research and represent the actual jobs professionals do every day.

The 25 Events include:

• The "Data Detective" (Data Cleaning): Finding missing values or spotting errors.
• The "Translator" (Table Join): Taking two separate lists and figuring out how they connect.
• The "Architect" (Table Transform): Taking a messy list and turning it into a neat, structured table.
• The "Coder" (NL-to-SQL): Turning a plain English question like "Show me top 5 sales" into the actual computer code (SQL) needed to get the answer.

The Results: The AI is Still a Rookie

The researchers tested the smartest AI models available (like OpenAI's GPT-5 and DeepSeek R1) on this "Olympics."

Here is the scorecard:

• The Best AI: Got about 69% correct.
• The "Reasoning" AI (models that think before they speak): Got about 58% correct.
• The Average AI: Struggled even more.

What does this mean?
Even the smartest AIs are still like rookie interns when it comes to complex table work. They can read a table, but if you ask them to reorganize a massive spreadsheet or fix a broken formula, they often make mistakes.

Key Discoveries (The "Aha!" Moments)

1. Thinking Helps, But Isn't Enough: Models that "think" (reasoning models) did better than standard chatbots, but they still struggled. It turns out, understanding a table requires more than just logic; it requires understanding the structure of the grid.
2. The "Long Table" Problem: If a table is short, the AI does fine. But if the table is huge (like a spreadsheet with thousands of rows), the AI gets lost. It's like trying to find a specific needle in a haystack, but the haystack is 10 miles wide. The AI forgets where it is.
3. The "Shuffle" Sensitivity: In a real spreadsheet, it doesn't matter if you shuffle the rows (the order of data). The meaning stays the same. But AI models get confused! If you shuffle the rows, the AI often thinks the data has changed. This shows they don't truly "understand" the table; they are just pattern-matching text.
4. Format Matters (Sometimes): AI used to get very confused if you gave them a table in a weird format (like HTML vs. CSV). Now, they are getting better at handling different formats, but they still prefer clean, simple layouts.

Why Should We Care?

We rely on tables for everything: banking, healthcare, logistics, and science. If we want AI to be a true "Copilot" for data experts (helping us build databases or analyze trends), it needs to pass the MMTU test.

The Bottom Line:
We have built a brilliant AI that can write poetry and solve calculus. But when it comes to the boring, messy, complex world of spreadsheets and databases, it's still a bit clumsy. MMTU is the new ruler we are using to measure how much it needs to grow up before it can truly help us with our data.

The authors hope that by publishing this benchmark, other researchers will build better models that can finally handle the "expert" level of table work, making our digital lives much easier.

Technical Summary

Here is a detailed technical summary of the paper "MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark."

1. Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in general language understanding and coding, their capabilities in expert-level table understanding and reasoning remain under-evaluated. Existing benchmarks are narrow, focusing primarily on:

• NL-to-SQL: Translating natural language to SQL queries.
• Table-QA: Answering questions based on table content.

These existing evaluations fail to capture the broad spectrum of real-world tasks faced by professional data engineers, analysts, and database administrators. Current models often struggle with complex operations involving table transformation, data cleaning, schema matching, and multi-table reasoning, which require a combination of coding, logical reasoning, and deep structural understanding of two-dimensional data.

2. Methodology: The MMTU Benchmark

The authors introduce MMTU (Massive Multi-Task Table Understanding and Reasoning), a large-scale benchmark designed to comprehensively evaluate foundation models on expert-level table tasks.

Data Curation & Composition

• Scale: The benchmark contains 28,136 questions derived from 61,763 real-world tables.
• Task Diversity: It covers 25 distinct task categories drawn from decades of research in data management, programming languages, and web data (e.g., SIGMOD, VLDB, PLDI).
• Task Categories:
  • Table Transform: Relationalization, output schema synthesis, and output table synthesis.
  • Table Matching: Entity matching, schema matching, and header-value matching.
  • Data Cleaning: Imputation, error detection, and list-to-table conversion.
  • Table Join: Semantic and equi-join detection.
  • Column Transform: Program transformation by example (Pandas/SQL), formula prediction, and semantic transformation.
  • Column Relationships: Identifying arithmetic, functional, and string relationships.
  • Table Understanding: "Needle-in-a-haystack" retrieval within tables.
  • Standard Tasks: NL-to-SQL, Table QA, Fact Verification, and Knowledge Base (KB) Mapping.
• Format: Questions follow a standardized triplet format: <Instruction, Input-Table(s), Ground-truth Answer>.
• Evaluation Framework: Unlike multiple-choice benchmarks (e.g., MMLU), MMTU uses structured, open-ended answers (JSON, SQL, Python code). It employs a custom evaluation framework supporting execution-based checks (for code) and structured output comparison (for JSON lists).

Quality Control

• Objective Selection: Tasks were selected based on having unique, manually labeled ground truths to ensure objective evaluation. Subjective tasks (e.g., table summarization) were excluded.
• Verification: An LLM-based quality check (using o4-mini) removed ambiguous or incorrect questions (~8%). Human domain experts verified a sample of 20 questions per task to ensure data integrity and instruction alignment.
• Privacy: Strict measures were taken to remove sensitive or private data.

3. Key Contributions

1. Comprehensive Benchmark: MMTU is the first benchmark to cover 25 diverse, expert-level table tasks, moving beyond the limited scope of NL-to-SQL and Table-QA.
2. Real-World Relevance: The tasks are derived from real-world professional workflows (e.g., data engineering, database administration) rather than synthetic or simplified datasets.
3. Evaluation Framework: The authors developed a flexible evaluation system capable of handling code generation, structured reasoning, and execution-based validation.
4. Open Access: The dataset and evaluation code are publicly released to foster further research.

4. Experimental Results

The authors evaluated a wide range of frontier models, including proprietary reasoning models (OpenAI GPT-5, o3, DeepSeek-R1) and chat models (GPT-5-Chat, DeepSeek-V3, Llama, Qwen).

Overall Performance

• Top Performance: Even the best-performing model, OpenAI GPT-5, achieved only 69.6% accuracy.
• Reasoning vs. Chat: Reasoning models significantly outperformed chat models (e.g., GPT-5: 69.6% vs. GPT-5-Chat: 57.7%). This highlights that complex table tasks require step-by-step reasoning and coding capabilities.
• Open Source Gap: While DeepSeek-R1 performed well (57.9%), a notable gap remains between top proprietary and open-source models.

Key Findings & Analysis

• Long Context Limitations: Performance degrades significantly as table size (rows/columns) increases. Models struggle with "Needle-in-a-Haystack" tasks in tables, particularly when retrieving data from the column direction (vertical reading), suggesting a bias toward 1D text processing.
• Robustness to Perturbation: Models are sensitive to row and column shuffling. While tables are semantically invariant to permutation, models often fail when the structural order changes, indicating a lack of robust understanding of table semantics.
• Format Sensitivity: Modern frontier models are becoming less sensitive to input formats (Markdown, CSV, JSON), though HTML remains a challenge. JSON often performs best for chat models due to easier value/column alignment.
• Error Analysis:
  • Table Understanding (38%): The largest error category, often involving misalignment of row/column indices or failure to maintain context in long tables.
  • Reasoning & Coding (28%): Errors in generating correct SQL/Pandas code or logical deductions.
  • Knowledge (18%): Hallucinations of facts or incorrect KB mappings.

5. Significance and Future Directions

• Benchmarking Standard: MMTU establishes a new standard for evaluating LLMs on structured data, moving the field toward general human-level intelligence in technical domains.
• Identifying Bottlenecks: The results clearly indicate that while LLMs are improving, they still lack robust 2D spatial reasoning and the ability to handle long-context table manipulation without hallucination or structural confusion.
• Driving Advancement: By highlighting specific failure modes (e.g., column shuffling sensitivity, long-context retrieval), MMTU provides a roadmap for developing better foundation models for data engineering, spreadsheet copilots, and database management systems.
• Future Work: The authors suggest extending the benchmark to include multi-modal inputs (visual table understanding) and incorporating more subjective/creative tasks like data generation and summarization.

In summary, MMTU reveals that despite rapid progress, current frontier models are not yet fully capable of expert-level table reasoning, necessitating further research into structural understanding and robust long-context processing.