Decomposition-Driven Multi-Table Retrieval and Reasoning for Numerical Question Answering

This paper proposes DMRAL, a decomposition-driven framework that constructs a table relationship graph and employs aligned question decomposition with coverage-aware retrieval and sub-question guided reasoning to significantly outperform existing methods in numerical multi-table question answering over large-scale table collections.

The Gist

Imagine you are a detective trying to solve a complex mystery: "How many total citations do all female Nobel Physics laureates have after 2010?"

To solve this, you don't just need one piece of paper. You need to dig through a massive, messy library containing 73,000 to 100,000 different spreadsheets (tables). Some are about Nobel prizes, some about scientists' genders, and some about their citation counts. Many of these spreadsheets are missing titles, have typos, or are split into tiny, confusing fragments.

Existing methods for solving these puzzles are like detectives who only know how to work in a neat, organized police station with a perfect filing system. When they walk into your messy library, they get lost, pick the wrong files, or give up.

This paper introduces a new detective team called DMRAL. Here is how they solve the case, explained with simple analogies:

1. The Problem: The "Messy Library"

Most current AI tools are trained on small, perfect databases (like a single police station). But the real world is a giant, chaotic data lake.

• The Scale: There are too many tables to read one by one.
• The Mess: Some files are missing labels (metadata), and the connections between files are hidden.
• The Complexity: To answer the question, you might need to join (connect) two files, union (stack) two similar files together, and then do math.

2. The Solution: The DMRAL Detective Team

The authors built a three-step system to handle this chaos.

Step A: The "Smart Question Breaker" (Table-Aligned Question Decomposer)

Instead of asking the AI, "Find the answer to this huge question," DMRAL breaks the question down into tiny, manageable clues, just like a detective breaking a big case into smaller leads.

• The Old Way: The AI guesses the sub-questions blindly. It might ask, "Who are the laureates?" and "Who is female?" but fail to realize it needs to look at specific columns in specific tables.
• The DMRAL Way: It looks at the library's structure first. It says, "Okay, to find 'female laureates,' I need to look at the 'Gender' column in the 'Nobel' table. To find 'citations,' I need the 'Citations' column." It aligns the clues with the actual shelves in the library before searching. This ensures no clue is missed and no time is wasted.

Step B: The "Coverage Detective" (Coverage-Aware Retriever)

Once the clues are broken down, the team needs to find the right files.

• The Old Way: They search for files that look somewhat related. They might grab a file about "Physics" but miss the specific file about "Post-2010," leading to an incomplete answer.
• The DMRAL Way: They use a "Coverage Score." Imagine a checklist. If the AI picks a file, it checks: "Does this file cover the 'Gender' clue? Does it cover the 'Year' clue?"
  • If the checklist isn't full, the AI doesn't stop. It asks, "What's missing?" and goes back to find a complementary file to fill the gap. It keeps searching until the entire question is covered by the selected files.

Step C: The "Step-by-Step Reasoner" (Sub-question Guided Reasoner)

Now that they have the right files, they need to do the math.

• The Old Way: The AI tries to write a giant, complex computer program (SQL) all at once to solve the whole puzzle. If it makes one small mistake (like a typo in a formula), the whole answer is wrong.
• The DMRAL Way: It builds the program like a ladder, one rung at a time.
  1. First, it writes a tiny program to find the female laureates.
  2. Then, it writes a second tiny program to get their citations.
  3. Finally, it combines them.
  • The Safety Net: After writing each step, it runs the code. If it crashes or gives an error, it immediately fixes the code before moving to the next step. This prevents small mistakes from ruining the final answer.

3. The Results: Why It Matters

The authors tested this system on two massive datasets they created (called SpiderWild and BirdWild), which simulate real-world messiness.

• Better Hunting: DMRAL found the right files 24% more often than the best existing methods.
• Better Answers: Because it found the right files and built the program carefully, it got the correct final number 55% more often.

The Big Picture

Think of existing AI as a brilliant student who can solve math problems perfectly if the teacher gives them a clean, single textbook.

DMRAL is like a seasoned field agent. It knows how to navigate a chaotic, messy archive, knows how to break a big problem into small tasks, knows how to double-check its work, and knows how to stitch together information from dozens of different sources to get the right answer.

This is a huge step forward for letting computers help us analyze the massive, messy data that exists in the real world today.

Technical Summary

Here is a detailed technical summary of the paper "Decomposition-Driven Multi-Table Retrieval and Reasoning for Numerical Question Answering" (DMRAL).

1. Problem Definition

The paper addresses Numerical Multi-Table Question Answering (MTQA) over large-scale table collections (e.g., web tables, data lakes, open data repositories).

• Context: Unlike traditional Text-to-SQL tasks which operate on small, well-defined relational databases with complete schemas and explicit Primary Key-Foreign Key (PK-FK) constraints, this task deals with massive, unstructured collections (tens of thousands of tables).
• Challenges:
  • Complex Relationships: Tables are related not just by joins (PK-FK) but also by unionability (tables with similar headers that can be stacked) and fuzzy joinability.
  • Incomplete Metadata: Column headers and titles are often missing or masked.
  • Scale: Retrieving relevant tables from tens of thousands of candidates is computationally difficult.
  • Numerical Complexity: Numerical questions require aggregation, arithmetic, and multi-step reasoning, leading to lower accuracy compared to textual QA.
• Limitations of Existing Methods:
  • Text-to-SQL: Fails due to lack of schema constraints and context limits.
  • Open-domain MTQA: Struggles with scale, ignores unionability, and suffers from error propagation when decomposing questions without table alignment.

2. Methodology: The DMRAL Framework

The authors propose DMRAL (Decomposition-driven Multi-table Retrieval and Answering), a framework consisting of four core modules:

A. Preprocessing: Table Relationship Graph

• Constructs a graph $G = (V, E)$ to capture complex inter-table relationships.
• Nodes ($V$): Clusters of unionable tables (tables with similar headers).
• Edges ($E$): Connections between clusters if any pair of tables is joinable (sharing overlapping or semantically similar values).
• This graph enables the system to navigate between joinable and unionable tables efficiently.

B. Table-Aligned Question Decomposer

Instead of blindly decomposing questions using an LLM, this module aligns the decomposition with the table structure:

1. Information Need Extraction: Parses the question to extract core concepts (entities, conditions).
2. Hybrid Column Matching: Uses M3-Embedding to match information needs against column snippets (title + header + sample values) to find candidate columns.
3. Context-Aware Disambiguation: Uses a greedy strategy on the Table Relationship Graph to select column mappings that ensure all selected tables form a connected component. This ensures the sub-questions are answerable within a coherent context.
4. Decomposition: Groups aligned information needs to generate specific sub-questions, ensuring Completeness (no missing info), Non-redundancy, and Table-specificity.

C. Coverage-Aware Retriever

This module retrieves the necessary tables to answer the decomposed sub-questions:

1. Learning-based Scoring: Uses a fine-tuned ColBERTv2 model to score candidate tables based on their semantic coverage of the sub-questions, filtering out superficial matches.
2. Coverage Verification:
   • Constructs Connected Table Groups (Steiner-tree inspired) where tables collectively cover all sub-questions.
   • Gap Detection: If a group's coverage score is low, the system generates a residual sub-question to identify missing "complementary tables," ensuring no information gaps exist.

D. Sub-question Guided Reasoner

Generates the final executable program (SQL or Python):

1. CoT-Guided Multi-step Generation: Instead of generating the whole program at once, it uses Chain-of-Thought (CoT) prompting. It generates a sub-program for each sub-question and incrementally joins them, respecting the dependencies identified in the decomposition phase.
2. Execution-Guided Refinement: The generated program is executed. If it fails (syntax error or logic error), the error message is fed back to the LLM to refine the program iteratively until a valid answer is produced.

3. Key Contributions

1. Novel Framework (DMRAL): The first framework specifically designed for numerical MTQA over large-scale, unstructured table collections, addressing the gap between Text-to-SQL and Open-domain QA.
2. Table Relationship Graph: A preprocessing pipeline that explicitly models unionability and joinability, allowing the system to handle complex table interdependencies.
3. Decomposition-Driven Retrieval: A strategy that aligns question decomposition with table structures (via column matching and graph connectivity) to prevent cascading retrieval errors.
4. Coverage-Aware Mechanism: A retrieval strategy that actively verifies and fills information gaps using residual sub-questions, ensuring complete table coverage.
5. New Benchmarks: Introduction of SpiderWild and BirdWild, two large-scale datasets containing ~73k and ~110k tables respectively, curated to simulate real-world conditions (incomplete metadata, complex relationships, and numerical reasoning).

4. Experimental Results

Experiments were conducted on SpiderWild and BirdWild, comparing DMRAL against state-of-the-art baselines (JAR, MMQA, OpenSearch-SQL, etc.).

• Table Retrieval: DMRAL achieved a 24% average improvement in retrieval effectiveness (F1 score) over existing methods. It significantly outperformed baselines in identifying relevant tables, especially for complex questions requiring unions and joins.
• Answer Accuracy: DMRAL achieved a 55% average improvement in answer accuracy (Exact Match) for numerical questions.
• Robustness:
  • Scale: Performance remained stable even as the table corpus size doubled (from 100k to 240k tables).
  • Metadata: DMRAL showed significantly less performance degradation (26% drop vs. 62% for baselines) when dealing with incomplete metadata.
  • Complexity: It handled "Hard" questions (requiring >2 joins, unions, and incomplete metadata) much better than baselines.
• Efficiency: While slightly slower than the fastest baselines due to the reasoning steps, it offered a strong trade-off between latency and accuracy, with retrieval times scaling linearly.

5. Significance

• Bridging the Gap: DMRAL solves the critical limitation of existing MTQA systems which fail when applied to the "messy" reality of large-scale web data and data lakes.
• Traceability: The decomposition-driven approach provides fine-grained traceability, allowing users to verify which tables were retrieved and how the reasoning was constructed, which is crucial for analytical applications.
• Scalability: The framework proves that complex numerical reasoning over massive, unstructured datasets is feasible without relying on pre-defined database schemas.
• Future Impact: The work establishes a new standard for evaluating MTQA (via SpiderWild/BirdWild) and provides a robust architecture for next-generation data analysis tools that interact with open data repositories.