TopBench: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

1. Problem Definition

While Large Language Models (LLMs) have advanced Table Question Answering (TQA), existing benchmarks primarily focus on explicit fact retrieval (finding a value in a row) or simple aggregation (calculating averages/sums). However, real-world tabular queries often involve implicit predictive reasoning, where the answer is not present in the data but must be inferred from historical patterns.

The paper identifies two core challenges in this domain:

1. Intent Recognition: Distinguishing between a request for data retrieval (e.g., "What is the charge for this user?") and a request for prediction (e.g., "What will the charge be for this new user profile?").
2. Predictive Reasoning: The ability to perform rigorous statistical modeling or reasoning over massive historical tables to infer unobserved outcomes, rather than relying on heuristic lookups or hallucinated retrieval.

Current benchmarks fail to evaluate these capabilities, as they lack datasets requiring the abstraction of unstructured intents into structured predictive tasks.

2. Methodology

A. The TOPBENCH Dataset

The authors introduce TOPBENCH, a benchmark comprising 779 high-quality samples derived from 35 real-world tables across three domains: Healthcare, Finance, and Daily Consulting. The dataset is structured into four distinct sub-tasks representing a hierarchy of predictive complexity:

1. Single-Point Prediction: Estimating a specific unknown value (regression/classification) for a new profile (e.g., predicting insurance costs for a specific patient).
2. Decision Making: Comparing multiple profiles to select an optimal solution based on predicted outcomes (e.g., "Which of these two patients will incur higher costs?").
3. Treatment Effect Analysis: Estimating the causal impact of changing state variables (counterfactual reasoning), such as predicting how a change in BMI or region affects future costs.
4. Ranking and Filtering: Screening massive candidate lists to identify top candidates based on implicit predictive criteria (e.g., "Find the top 3 candidates with the highest predicted risk").

Data Construction:

• Logic-Driven Sampling: Instead of random selection, the authors use "Hard Negative Sampling" to create queries with similar feature values or high noise, forcing models to rely on precise modeling rather than rough heuristics.
• Dual-Perspective Prompting: Queries are generated from two personas: a non-technical "User" (natural language, narrative) and a "Data Holder" (expert, business-centric), ensuring diverse intent expression.
• Validation: A hybrid pipeline involving LLM auditors and human experts ensures solvability and logical consistency.

B. Evaluation Framework

TOPBENCH employs a dual-stream evaluation methodology:

1. Natural Language Reasoning (Tasks 1-3): Uses an LLM-as-a-Judge pipeline.
   • Extraction: A specialized judge extracts predicted values, confidence intervals, and proof quotes from free-form responses.
   • Hallucination Verification: A multi-stage verification process (surface matching, numeric parsing, and NLI) ensures extracted values are strictly grounded in the model's output.
   • Metrics: Combines NMAE (Normalized Mean Absolute Error) with a Logic Score (0-5) to penalize circular reasoning or self-contradiction.
2. Structured Output (Task 4): Uses deterministic file-based evaluation.
   • Models must generate CSV files.
   • Metrics include F1 Score (for filtering), NDCG (for ranking quality), and Batch NMAE (for prediction precision).

C. Experimental Setup

The authors evaluated 9 models across three categories:

• General LLMs: GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, DeepSeek-V3.2, Qwen3.
• Reasoning-Enhanced Models: DeepSeek-V3.2-Thinking, Qwen3-Thinking.
• Tabular Specialists: TableLLM-8B/13B.

Two inference paradigms were tested:

1. Text-Based Reasoning: Direct prompting with truncated table data.
2. Agentic Workflow: Iterative ReAct loops where models write and execute Python code in a sandbox to perform data processing and modeling.

3. Key Results

A. Performance Gap

• Low Accuracy: Current LLMs struggle significantly with implicit prediction. Most models score below 0.60 on accuracy metrics, a sharp contrast to their proficiency in fact retrieval.
• Retrieval vs. Prediction: Models frequently default to heuristic retrieval (finding the "nearest neighbor" in the truncated context) rather than building predictive models. This leads to "hallucinated retrieval" where the model claims to find a match that doesn't exist.
• Reasoning Models: Surprisingly, "Thinking" models (e.g., Qwen3-Thinking) showed instability in text-based settings, often entering repetitive loops that exhausted context windows without solving the task.

B. Impact of Agentic Frameworks

• Mixed Results: While code execution helps with complex arithmetic (improving Trend Scores in Treatment Effect Analysis), it does not guarantee predictive success.
• Intent Misalignment: In the agentic setting, models like Qwen3-Instruct often failed to switch from "retrieval mode" to "modeling mode," attempting to filter data via code rather than training a predictor, leading to performance drops.
• Tool Usage: Models that actively invoked machine learning libraries (e.g., scikit-learn Random Forest) significantly outperformed those relying on simple pandas filtering or arithmetic.

C. Ablation Studies

• Intent Recognition is Prerequisite: Providing explicit semantic hints (e.g., specifying the target column or task type) significantly improved performance (e.g., Qwen3-Instruct's Single-Point accuracy rose from 0.43 to 0.56). This confirms that intent disambiguation is the primary bottleneck.
• Modeling Capacity: Even with correct intent, simple algorithms (like unoptimized Linear Regression) often failed to capture complex non-linear dependencies. A "Predict-Only" baseline using an ensemble of advanced tabular models (XGBoost, TabPFN) achieved significantly higher upper bounds (e.g., 0.76 vs. 0.66), indicating that sophisticated feature engineering and model selection are required beyond just intent recognition.

4. Key Contributions

1. Formal Definition: The first formal definition and quantification of Implicit Predictive TQA, bridging the gap between standard TQA and real-world forecasting.
2. TOPBENCH Benchmark: A new standard dataset with 779 samples across four task types, supporting rigorous automated evaluation via a hybrid LLM-as-a-Judge and deterministic pipeline.
3. Comprehensive Analysis: A systematic failure analysis revealing that current LLMs lack the ability to distinguish retrieval from prediction intents and struggle with the full pipeline of data preprocessing, model selection, and rigorous reasoning.

5. Significance

• Paradigm Shift: TOPBENCH shifts the TQA paradigm from information retrieval to predictive intelligence, addressing a critical capability gap in industrial applications (e.g., risk assessment, financial forecasting).
• Diagnostic Tool: The benchmark reveals that current "Reasoning" models often fail due to intent misalignment rather than a lack of reasoning depth. It highlights the need for models to autonomously recognize when to switch from "searching" to "modeling."
• Future Directions: The paper suggests that future autonomous data intelligence systems must integrate intent-rich query understanding with robust, adaptive tabular modeling pipelines (including feature engineering and model selection) to handle unstructured real-world requests effectively.

In summary, TOPBENCH demonstrates that while LLMs are excellent at retrieving facts, they currently lack the robustness to perform predictive reasoning over tabular data without significant guidance, highlighting a major frontier for future research in tabular foundation models and agentic workflows.