Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

1. Problem Definition: Tabular-Vision Multi-Modal Understanding (TVMU)

The paper addresses Tabular-Vision Multi-Modal Understanding (TVMU), a task where a model must reason over tables extracted from images (document images, scanned copies) combined with textual context and visual cues. Unlike standard table QA where the full table is provided as text, TVMU requires the model to:

1. Parse table structures from visual inputs (which may be incomplete, merged, or poorly formatted).
2. Integrate visual information (e.g., pet images in a dataset) with tabular data.
3. Perform complex reasoning, including information extraction, mathematical computation, and predictive modeling.

The authors identify three core challenges in existing approaches:

• Structural Variability & Data Incompleteness: Real-world tables vary widely (e.g., three-line tables, merged cells) and often contain missing data or OCR errors. Existing models struggle when the full table is not explicitly provided as text.
• Implicit & Complex Feature Dependencies: Models often fail to capture the interdependencies between features and prediction targets, especially when these relationships are not explicitly stated but must be inferred through computation.
• Task Heterogeneity: TVMU encompasses diverse tasks like Multimodal Table Question Answering (QA) and Multimodal Table Prediction (classification/regression). Existing methods are often designed for a single task type and lack a unified framework to handle both.

2. Methodology: Thinking with Tables (TWT)

The authors propose Thinking with Tables (TWT), a program-aided neuro-symbolic reasoning framework. Instead of relying solely on end-to-end neural prediction, TWT treats table understanding as an interactive reasoning process within an external sandbox environment.

Core Mechanism

TWT operates via a strict interaction protocol involving four components:

1. <analy>: The model explicitly reasons about the problem and plans steps.
2. <code>: The model generates Python code to interact with the external environment (reading CSVs, loading images, calling pre-trained models).
3. <code_results>: A sandbox executes the code and returns results or error messages.
4. <answer>: The model synthesizes the results to produce the final answer.

This approach allows the model to offload complex computations and data manipulation to code, ensuring robustness against structural variability and enabling precise feature modeling.

Two-Stage Training Strategy

To train TWT effectively, the authors employ a two-stage pipeline:

Stage 1: Task-Oriented Supervised Fine-Tuning (TO-SFT)

• Goal: Teach the model the problem-solving pipelines for different tasks (QA vs. Prediction) and how to interact with the sandbox.
• Data Synthesis: Since existing datasets lack reasoning traces, the authors synthesize high-quality training data using a stronger MLLM (Qwen3-VL-Plus) and an LLM. The MLLM handles visual understanding and step generation, while the LLM refines the code and computation.
• Training: The model is fine-tuned on these synthesized trajectories. Crucially, code execution outputs are masked during loss calculation to prevent the model from memorizing results rather than learning the reasoning process.

Stage 2: Adaptive Loss-Scaled Reinforcement Learning (AL-GRPO)

• Goal: Enhance reasoning capabilities, stability, and adaptability to complex scenarios.
• Reward Design: Unlike standard GRPO which focuses on format, this stage focuses on answer correctness.
  • For regression tasks, the reward is based on the relative error between the predicted and ground-truth values.
  • For classification/QA, it uses binary correctness.
• Adaptive Loss Scaling: To handle code execution failures (e.g., infinite loops, syntax errors), the authors introduce a dynamic strategy. Only code segments that execute successfully (sandbox output indicates normal exit) contribute to the policy gradient update. This prevents the model from being penalized for exploratory failures while encouraging stable, executable code.

3. Key Contributions

1. Problem Formulation: A systematic analysis of TVMU, highlighting the specific challenges of structural variability, incomplete data, and task heterogeneity in real-world multimodal settings.
2. Novel Framework (TWT): Introduction of a neuro-symbolic framework that uses code-based interactive reasoning to bridge the gap between visual table parsing and complex logical/computational reasoning.
3. Unified Training Strategy: A two-stage approach (TO-SFT + AL-GRPO) that successfully unifies diverse downstream tasks (QA and Prediction) under a single model, addressing the heterogeneity challenge.
4. Data Synthesis Pipeline: A verifiable method for generating high-quality reasoning traces with code execution feedback, overcoming the lack of reasoning annotations in existing TVMU datasets.

4. Experimental Results

The authors evaluated TWT on eight representative datasets covering both QA (WikiTQ, TabMWP, FinQA, TAT-QA) and Prediction (Adoption, SkinCA, Pawpularity, Paintings).

• Performance: TWT consistently outperformed existing baselines (including open-source MLLMs like Qwen3-VL and InternVL, and specialized table models like HIPPO and Table-LLaVA).
  • It achieved an average accuracy improvement of ~10% over the strongest open-source baselines.
  • On several tasks, TWT matched or surpassed proprietary commercial SOTA models (e.g., GPT-5.1, Qwen3-VL-Plus).
• Robustness:
  • Incomplete Data: TWT maintained high performance even when table features were missing from the visual input, relying on code to extract and model data dynamically.
  • Code Execution: The AL-GRPO stage significantly improved code execution accuracy, ensuring the model could reliably interact with the sandbox.
• Ablation Studies: Confirmed that both the TO-SFT stage (learning pipelines) and the AL-GRPO stage (refining reasoning and stability) are essential for the final performance.

5. Significance

• Bridging the Gap: TWT demonstrates that neuro-symbolic reasoning (combining neural perception with symbolic code execution) is highly effective for tabular data, a domain where pure neural models often struggle with precision and logical consistency.
• Real-World Applicability: By handling incomplete data and diverse table structures through code interaction, TWT moves beyond synthetic benchmarks to address the messy reality of financial reports, medical records, and industrial data.
• Unified Architecture: The work proves that a single framework can effectively handle both question-answering and predictive modeling tasks, suggesting a path toward more generalizable multimodal agents.

In conclusion, Thinking with Tables establishes a new paradigm for tabular understanding, leveraging the precision of code execution to enhance the reasoning capabilities of Large Multimodal Models, resulting in state-of-the-art performance across complex, real-world multimodal scenarios.