Technical Summary: ODTQA-FoRe and TimeFore

Problem Definition

Despite the rapid advancement of Large Language Models (LLMs) in Tabular Question Answering (TQA), existing systems predominantly focus on retrieving and reasoning over static historical data. They lack the capability to perform future-oriented numerical prediction and forecast-based reasoning. Current open-domain TQA methods struggle with two primary limitations: (1) LLMs are trained on static text and lack mechanisms to model dynamic time-series data effectively, and (2) in open-domain scenarios, systems cannot rely on pre-provided continuous historical data; they must autonomously retrieve relevant data from massive databases to support forecasting.

To address this gap, the authors introduce a novel task: Open-Domain Tabular Question Answering for Future Data Forecasting and Reasoning (ODTQA-FoRe). This task requires a system to:

1. Autonomously identify and retrieve relevant tables from a large corpus of candidate tables.
2. Perform time-series forecasting on the retrieved historical data.
3. Synthesize forecasts to answer complex queries, including direct future predictions and comparative reasoning based on those forecasts.

Methodology

1. Dataset Construction: ODTQA-FoRe

The authors constructed the first benchmark dataset for this task using real estate transaction data from 10 Chinese cities spanning 2022–2024.

• Scale: The dataset contains 28,507 QA pairs derived from 11,149 real estate projects.
• Structure: It comprises 288 tables (averaging 845 rows each) representing historical data (2022–2023) and a separate "future" database (2024) used for ground truth evaluation.
• Task Distribution: The dataset includes 8,042 time-series forecasting questions (e.g., "What will the price be next month?") and 20,465 forecast-based reasoning questions (e.g., "Which project will have a larger price increase?").
• Generation: Questions were generated using 26 seed templates covering single and multi-project scenarios. To enhance linguistic diversity and realism, an LLM (Qwen2.5-72B) was used to rewrite template-generated queries, followed by manual validation.
• Split: Data is partitioned at the project level (6:2:2 ratio) into training, validation, and test sets to prevent data leakage.

2. Proposed Framework: TimeFore

To solve the challenges of ODTQA-FoRe, the authors propose TimeFore, an LLM agent-based framework that decomposes the problem into three collaborative roles:

• Retriever:

  • Function: Identifies relevant tables and generates SQL queries to fetch historical data.
  • Mechanism: It employs a two-stage retrieval process. First, an LLM summarizes the user query into a canonical table caption using few-shot prompting. If a direct match fails, the BM25 algorithm retrieves the most semantically relevant table.
  • SQL Generation: The agent generates SQL queries using in-context learning (5-shot) and employs an execution-feedback loop. It executes the SQL via a sqlQueryTool; if errors occur, it refines the query based on the error message, repeating up to 25 times.

• Forecaster:

  • Function: Addresses the LLM's inherent weakness in time-series prediction by invoking specialized external models.
  • Mechanism: The agent receives serialized historical data (project name, month, price) and formats it into a 24-month numerical list. It then calls a predefined function, imputationThenPredictionTool, which orchestrates a pipeline:
    1. Imputation: Uses TimesNet to handle missing values in the historical sequence.
    2. Prediction: Uses TimeXer to forecast the next 12 months of prices.
  • Output: Returns a structured list of predicted prices for 2024.

• Analyzer:

  • Function: Synthesizes historical data and forecasts to generate the final standardized answer.
  • Mechanism: It first classifies the query type (direct forecasting vs. forecast-based reasoning) using a BERT-based classifier.
  • Reasoning: Based on the classification, it selects a specific prompt (in-context learning with 5 examples) to guide the LLM.
  • Normalization: A dedicated numerical extraction module ensures the output is concise and strictly formatted (e.g., extracting only the required number or list of projects), preventing verbose explanations common in LLMs.

Key Contributions

1. Novel Task Definition: The paper defines ODTQA-FoRe, expanding the scope of LLM-based QA to include future data prediction and reasoning.
2. Benchmark Dataset: The introduction of ODTQA-FoRe, the first dataset covering time-series forecasting and forecast-based reasoning in an open-domain tabular setting, filling a critical resource gap.
3. Framework Proposal: The development of TimeFore, a collaborative agent framework that effectively bridges the gap between LLMs and specialized time-series models, addressing retrieval, forecasting, and reasoning challenges.
4. Empirical Validation: Comprehensive experiments establishing strong performance benchmarks across five mainstream LLMs (Qwen3, GPT-OSS, GLM) and demonstrating the necessity of hybrid approaches.

Experimental Results

The authors evaluated TimeFore against a "Vanilla" baseline (LLM-only with BM25 retrieval and in-context learning) and various standalone models.

• Forecasting Performance: Specialized time-series models significantly outperformed general-purpose LLMs. TimeXer achieved the best performance (MRE: 0.1209) compared to general LLMs like Qwen3 30B (MRE: 0.1706) and GLM 4.5 Air (MRE: 0.1869).
• End-to-End Performance: TimeFore significantly outperformed the Vanilla baseline across all metrics. For example, on the Qwen3 30B model, TimeFore improved Forecast-based Reasoning F1-score from 24.87% to 60.25% and reduced Forecasting MSE by approximately 22%.
• Ablation Studies:
  • Forecasting as the Bottleneck: The study revealed that the accuracy of the time-series forecasting component is the primary bottleneck. Replacing predicted future data with ground truth (Golden Future) yielded the most substantial performance gains (e.g., +46.80% accuracy for Qwen3 30B), whereas improvements from perfect table retrieval or SQL generation were marginal.
  • Retriever Robustness: The two-stage retrieval method (Summary + BM25) achieved high F1 scores (>97%), confirming that data acquisition is robust and not the primary source of error.
  • Analyzer Necessity: The numerical extraction module in the Analyzer was found crucial; removing it caused a significant drop in the Valid Completion Rate (e.g., a 33.08% drop for Qwen3 30B).

Significance and Claims

The paper claims that ODTQA-FoRe and TimeFore represent a foundational step toward enabling LLMs to handle dynamic, future-oriented decision-making tasks. The authors emphasize that:

• Hybrid Architecture is Essential: Relying solely on LLMs for time-series prediction is insufficient. Delegating forecasting to specialized models while leveraging LLMs for orchestration, retrieval, and reasoning yields a more robust framework.
• Benchmarking Gap: The work provides the first standardized benchmark for open-domain tabular forecasting, enabling future research to evaluate and improve upon these capabilities.
• Real-World Applicability: By focusing on the real estate domain, the study demonstrates the practical utility of such systems for investment and financial decision-making, though the framework is designed to be domain-agnostic.

The authors modestly note limitations, including the current restriction to the real estate domain and the lack of integration with exogenous variables (e.g., macroeconomic policies), identifying these as directions for future work.