Technical Summary: Reasoning by Commented Code for Table Question Answering

Problem Statement

Table Question Answering (TableQA) presents a significant challenge for Large Language Models (LLMs) due to the fundamental mismatch between the linear token processing of autoregressive models and the two-dimensional structure of tabular data. While LLMs excel in natural language understanding, they struggle with structured data reasoning. Existing approaches face specific limitations:

• Text-to-SQL: Often assumes homogeneous column data types, failing when real-world tables contain mixed data (e.g., numbers with symbols or annotations).
• End-to-End Models: Rely on implicit token generation for arithmetic, leading to numerical inaccuracies. They also suffer from scalability issues and the "lost-in-the-middle" phenomenon as table size increases.
• Program-Based Methods (Single-Line): While offering better numerical reliability via code execution, previous methods (e.g., Repanda) often compress reasoning into single-line expressions without comments. This lack of explicit intermediate steps hinders interpretability and error analysis.

Methodology

The paper proposes a commented, step-by-step code-generation framework that grounds TableQA reasoning directly in executable Python (Pandas) code. The core innovation is the integration of explicit natural language reasoning comments within the code generation process itself.

1. Structured Instruction and Output

The model is instructed to generate a single Python function where every major Pandas operation is preceded by a structured reasoning comment. The output format is strictly standardized:

• # PLAN: A mandatory high-level summary of the strategy.
• Intermediate Steps: Optional but necessary comments (e.g., # FILTER, # PARSING, # AGGREGATE) that explain the specific dataframe transformation being performed.
• # ANSWER: A comment summarizing the final result before the return statement.
• Robustness Constraints: The instructions explicitly guide the model to handle noisy data (e.g., converting string-formatted numbers using pd.to_numeric(..., errors='coerce'), parsing flexible dates, and ignoring summary rows like "Total" unless required).

2. Training Data Construction

Since the WikiTableQuestions (WikiTQ) dataset lacks intermediate reasoning traces, the authors constructed a training dataset using a self-improvement pipeline:

• Generation: A powerful model (Qwen2.5-Coder-32B-Instruct) generates commented Pandas solutions for table-question pairs.
• Error-Guided Refinement: If the generated code fails execution or produces an incorrect answer, the model is prompted with the error, the golden answer, and the original input to regenerate a corrected solution.
• Filtering: Solutions that merely memorize the golden answer without valid code logic are excluded.

3. Answer Selection Mechanism

To leverage the complementary strengths of code-based execution and end-to-end reasoning, the authors introduce a lightweight Answer Selector.

• For a given table and question, two candidates are generated: one from the proposed code-based method and one from an end-to-end model (Table-R1).
• A small selector model (e.g., Qwen3-4B) is trained to choose the correct candidate based solely on the input table and question, without generating new answers.

Key Contributions

1. Commented Reasoning Framework: A novel approach that aligns step-wise table reasoning with executable Pandas code, making the reasoning process interpretable and reliable.
2. Structured Instruction Design: A unified output format that enforces explicit planning, instruction following, and robust data handling within a single-pass program generation setting.
3. Hybrid Integration: A demonstration that grounding reasoning in executable code improves numerical reliability and can be effectively combined with end-to-end models via an answer selector to achieve state-of-the-art performance.

Experimental Results

The framework was evaluated on the WikiTableQuestions (WikiTQ) benchmark using Fuzzy Match (FM) accuracy as the primary metric.

• Base Performance: The proposed method using Qwen2.5-Coder-7B-Instruct achieved 70.9% accuracy, surpassing the Repanda baseline (67.6%) which uses a similar model scale but lacks explicit reasoning comments.
• Scaling: Performance improved with model size, with Qwen2.5-Coder-14B-Instruct reaching 72.1%.
• Combined System: By integrating the code-based method with the end-to-end model Table-R1 (which achieves 81.7% alone) via the answer selector, the system achieved a peak accuracy of 84.3%.
• Error Analysis: The paper notes that the code-based approach excels in numerical aggregation tasks where end-to-end models fail (e.g., calculating averages), while end-to-end models sometimes succeed where code generation fails due to empty cells or complex reasoning instability. The selector effectively mitigates these complementary failure modes.

Significance and Claims

The paper claims that grounding table reasoning in step-wise, commented code provides a scalable foundation for handling intensive numerical tasks and unstructured table formats.

• Interpretability: Unlike opaque single-line queries, the proposed method makes intermediate decisions explicit, facilitating debugging and error analysis.
• Numerical Exactness: By delegating computation to a symbolic execution engine (Pandas), the method ensures deterministic numerical results, overcoming the approximation errors of token-based arithmetic.
• Practicality: The approach narrows the performance gap for smaller, open-source models, offering a transparent and verifiable mechanism for real-world TableQA applications without requiring massive proprietary models.

The authors acknowledge limitations, noting that the evaluation is primarily on WikiTQ, a benchmark well-suited for executable reasoning, and that the method may be less necessary for very small tables requiring minimal computation. They also highlight that dataset noise (annotation errors in WikiTQ) remains a factor in residual errors.