TabularMath: Understanding Math Reasoning over Tables with Large Language Models

1. Problem Statement

While Large Language Models (LLMs) have shown significant progress in solving mathematical word problems (MWP) presented as plain text, their ability to perform mathematical reasoning over tabular data remains underexplored and poorly evaluated. Real-world applications (e.g., business intelligence, financial forecasting) require models to:

• Navigate highly structured, semi-structured data.
• Perform multi-step numerical reasoning involving cross-column computations.
• Handle imperfect data, such as missing values or contradictory information, which is common in real-world scenarios but often ignored in existing benchmarks.

Current benchmarks suffer from three main limitations:

1. Scalability: They rely heavily on manual annotation, making it difficult to generate large-scale, diverse datasets.
2. Complexity: They often use simple tables that do not challenge the retrieval and reasoning capabilities of advanced models.
3. Robustness: They fail to evaluate how models behave when faced with incomplete or inconsistent data, overlooking the risk of hallucinations.

2. Methodology

The authors propose a two-pronged approach: a neuro-symbolic framework for data generation and a comprehensive benchmark for evaluation.

A. AUTOT2T: Automatic Text-to-Table Framework

To overcome the scalability and annotation bottleneck, the authors introduce AUTOT2T, a neuro-symbolic pipeline that converts existing math word problems into scalable, verified tabular reasoning tasks without human intervention. The pipeline consists of three stages:

1. Semantic Decoupling:

   • The system uses an LLM to decompose a math word problem into a formal logical representation (using SMT-Lib syntax).
   • It extracts variables ($V$), constraints ($C$), and expressions ($e$).
   • A formal solver (e.g., Z3, CVC5) verifies the satisfiability and consistency of the extracted state, filtering out ill-defined problems.

2. Table Transformation:

   • The formal state is converted into a structured table. The LLM maps variables and active constraints to table columns, using a "name" field as the primary key.
   • A formal solver validates the generated table against the original constraints to ensure entity extraction correctness.

3. Table Augmentation:

   • Starting from a "seed" table, the system applies randomized operations to create diverse variants:
     • RowAug: Adds rows with different entities (simulating larger datasets).
     • ColAug: Adds irrelevant columns to increase retrieval noise.
     • OrdShf: Shuffles row/column orders to hinder pattern matching.
     • InfMut: Modifies information to create Imperfect scenarios (missing data or contradictory constraints).

B. TabularMath Benchmark

Built upon AUTOT2T, TabularMath is a comprehensive benchmark containing 6,782 samples (both text and image modalities). It comprises four subsets:

• Easy, Medium, Hard: Progressively increasing table complexity (measured by row/column count and structural noise) to test retrieval and reasoning limits.
• Imperfect: Contains 1,000 instances with either Missing information (key data removed) or Contradictory information (logical conflicts injected) to test robustness.

3. Key Contributions

1. AUTOT2T Framework: A novel neuro-symbolic pipeline that enables the automatic, controllable, and verified generation of large-scale tabular math reasoning datasets, addressing the lack of scalable data creation methods.
2. TabularMath Benchmark: The first benchmark to systematically evaluate LLMs on tabular math reasoning across complexity, quality (robustness), and representation (text vs. image) dimensions.
3. Comprehensive Evaluation: Systematic testing of 18 models (open-source general, math-specialized, table-specialized, and closed-source APIs) revealing critical gaps in current capabilities.

4. Key Results and Observations

The study yields three primary observations regarding model performance:

Observation 1: The Coupling of Retrieval and Reasoning

• Performance Drop: All models suffer significant performance degradation when transitioning from pure text (GSM8K) to tabular modalities.
• Complexity Impact: As table complexity increases (from Easy to Hard), accuracy drops sharply.
• Retrieval vs. Reasoning Gap: Models perform significantly better on pure retrieval tasks (finding a specific value) than on joint reasoning tasks. The performance gap exceeds 20% on average, indicating that the core bottleneck is not just finding data, but integrating retrieval with multi-step reasoning.

Observation 2: Vulnerability to Low-Quality Data

• Hallucination Risk: When tables contain missing or contradictory information, most models fail to detect the flaw. Instead of abstaining, they generate misleading answers.
• Error Rates: In some cases, error rates on imperfect data exceed 50%.
• Trade-off: When explicitly warned about potential traps, models tend to reject valid problems more often, demonstrating a trade-off between solvency (solving valid problems) and discriminative ability (detecting invalid ones).

Observation 3: Modality and Format Effects

• Text vs. Image: Text-based tables are consistently easier for models to reason over than image-based tables, even for Multimodal LLMs (MLLMs). Image tables introduce OCR noise and layout ambiguity.
• Format Matters: Among text formats, JSON and Serialized (key-value) formats outperform Markdown. The explicit key-value structure in JSON facilitates better retrieval, while Markdown's parsing ambiguity degrades performance on complex tables.

5. Significance and Future Directions

• Real-World Applicability: The work highlights that current LLMs are not yet robust enough for critical real-world applications (e.g., financial analysis) where data is often messy or incomplete.
• Research Guidance: The findings suggest that future research should focus on:
  • Improving the retrieval-reasoning coupling (helping models know what to retrieve and how to use it).
  • Developing mechanisms for uncertainty quantification and refusal when data is insufficient.
  • Optimizing data representation (favoring structured formats like JSON over unstructured text or images).
• Data Generation: The AUTOT2T framework provides a blueprint for generating high-quality, verified datasets for other structured reasoning tasks, moving beyond manual curation.

In conclusion, TabularMath establishes a new standard for evaluating LLMs on structured data, revealing that while models are improving, they still struggle with the complexity, noise, and imperfections inherent in real-world tabular reasoning.