Technical Summary: LAKEQA – An Exploratory QA Benchmark over a Million-Scale Data Lake

Problem Definition

Recent Large Language Models (LLMs) have demonstrated significant progress in reading-based Question Answering (QA), where evidence is either explicitly provided or trivially retrievable from narrow, curated corpora. However, real-world scenarios often involve Exploratory Question Answering (EQA) over heterogeneous data lakes. In these settings, evidence is not provided upfront; instead, it is scattered across massive collections of structured (e.g., CSV, JSON) and unstructured (e.g., PDF, HTML) documents. Solving such problems requires agents to iteratively search for relevant files, discover intermediate entities, and reason across multiple dependent steps before formulating a final answer.

Current benchmarks fail to capture this complexity simultaneously. Existing datasets either possess high search intensity (large document collections) but low reasoning depth (few hops), or high reasoning depth but operate over small, curated search spaces. There is a lack of a comprehensive benchmark that demands both high search intensity (scanning millions of documents) and high reasoning intensity (long-horizon, multi-hop reasoning with implicit intermediate steps).

Methodology

1. Benchmark Construction (LAKEQA)

The authors introduce LAKEQA, a benchmark designed to evaluate EQA capabilities over a massive, heterogeneous data lake.

• Data Lake Scale: The benchmark is built upon a collection of approximately 9.5 TB of data, comprising roughly 40 million unique documents.
• Sources: The data is aggregated from two complementary open-data sources:
  • Wikipedia: Parsed from the latest English dump to serve as a canonical text corpus.
  • Data.gov: Harvested from the Harvard LIL data.gov archive, providing diverse, weakly standardized public government artifacts (structured tables, metadata, reports).
• Task Design: Each task is a complex natural language question requiring an agent to:
  1. Identify relevant documents from the 40M collection.
  2. Extract supporting facts.
  3. Chain reasoning steps where the output of one step serves as the input condition for the next.
• Annotation Process: Tasks are constructed via a human-in-the-loop process involving Ph.D. level experts and senior undergraduates. The process involves:
  • Subquestion Construction: Annotators derive facts from documents and formulate sub-questions, chaining them such that $Answer_i$ is a prerequisite for $Question_{i+1}$.
  • Question Rewrite: The chain is rewritten into a single, natural-sounding question that hides document identifiers to prevent trivial discovery.
  • Quality Control: Each task undergoes multi-stage review by independent annotators to ensure logical validity, naturalness, and that the final answer is uniquely determined by the evidence.

2. Experimental Setup

• Models Evaluated: Seven frontier LLMs were tested, including proprietary models (OpenAI GPT-5.2, GPT-5-mini, Anthropic Claude Haiku/Sonnet/Opus 4.5) and open-source models (DeepSeek-R1, Llama-3.3-70B).
• Agent Environment: Models were equipped with a tool suite mimicking realistic data analysis workflows, including:
  • search(query): Keyword and semantic search over the data lake.
  • list_files, download, inspect: File management and preview.
  • execute_code: Python execution for local data analysis (filtering, aggregation).
• Baselines: Retrieval-Augmented Generation (RAG) baselines were tested using BM25 and Hybrid (BM25 + Vector) search over sampled subsets of the data lake (10k–50k documents).
• Metrics: Performance was measured using Exact Match (EM) for end-to-end accuracy, alongside document-level Precision, Recall, and F1 for search effectiveness.

Key Results

1. Performance Gap

Even with access to search and inspection tools, state-of-the-art LLMs struggle significantly with LAKEQA:

• Low Accuracy: The best-performing model, Claude-sonnet-4.5, achieved only 32.87% Exact Match on the full benchmark (LAKEQA-full). GPT-5.2 scored 18.37%, and open-source models performed even lower.
• Cost-Accuracy Trade-off: While larger models (e.g., Claude Opus) showed higher accuracy on the mini-set (45.93%), they incurred significantly higher costs and runtime. Smaller, cheaper models often failed due to inefficient code generation or poor reasoning.

2. Failure Analysis

Trajectory analysis revealed that the primary bottleneck is document discovery, not reasoning over found evidence:

• Search Failure: Models frequently failed to retrieve the "gold" documents required to answer the question. Recall on the retrieved set ($D_{ret}$) was low, indicating agents could not surface relevant files.
• Exploration Failure: Even when relevant documents were retrieved, agents often failed to inspect or analyze them. For instance, on LAKEQA-full, Claude-sonnet-4.5 achieved 42.36% recall on retrieved documents but only 13.04% recall on accessed documents ($D_{acc}$), suggesting a disconnect between retrieval and effective exploration.
• Error Accumulation: Performance degraded sharply as the number of reasoning hops increased. Tasks requiring >18 hops saw a drastic drop in accuracy, indicating that long-horizon exploration amplifies compounding errors from imperfect retrieval and missed constraints.

3. RAG Baselines

Adding retrieval tools to reduced document subsets (10k–50k) improved performance slightly but did not solve the problem. The results suggest that EQA is not a single-shot retrieval task; agents must iteratively infer missing information, formulate new queries, and decide when evidence is sufficient, a capability current RAG systems lack.

Significance and Claims

The paper positions LAKEQA as a realistic testbed for developing LLM agents capable of operating in modern data lake environments. Its significance lies in:

1. Bridging the Gap: It is the first benchmark to simultaneously scale both search intensity (40M documents) and reasoning intensity (multi-hop chains), addressing a limitation in prior works like OTT-QA (high search, low reasoning) and MuSiQue (high reasoning, low search).
2. Identifying Bottlenecks: The experiments demonstrate that current LLM agents are not yet capable of effective iterative search and exploration. The primary failure mode is the inability to discover and open the correct evidence documents, rather than the inability to reason once evidence is found.
3. Real-World Relevance: By utilizing heterogeneous government and Wikipedia data, LAKEQA reflects the messy, unstructured reality of real-world data lakes, moving beyond curated corpora to challenge agents with the complexity of discovering evidence in the dark.

The authors conclude that while LLMs have advanced in reading comprehension, effective search and exploration remain the primary bottlenecks for solving complex, open-domain questions over large-scale data lakes.