Technical Summary: LAKEQUEST

Problem Definition

Modern Question Answering (QA) systems have achieved significant success on curated, homogeneous corpora (e.g., reading comprehension, open-domain text retrieval). However, real-world enterprise and scientific knowledge is rarely stored as flat, schema-aligned text. Instead, it resides in data lakes: heterogeneous, weakly structured collections of tables, passages, and linked metadata.

Current benchmarks fail to evaluate the end-to-end performance required to navigate these environments. They either abstract away the noisy discovery process or evaluate reasoning over already-clean data. Consequently, there is a lack of evaluation environments that simultaneously stress source discovery, cross-source composition, and faithful attribution over heterogeneous, weakly structured corpora.

Methodology: The LAKEQUEST Framework

To address this gap, the authors introduce LAKEQUEST, a human-validated benchmark designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes.

1. Dataset Construction

LAKEQUEST spans three diverse domains, preserving their native structural heterogeneity rather than flattening them into a uniform representation:

• AI/ML Metadata: A repository of model and dataset cards from the Hugging Face Hub, focusing on entity disambiguation and relation chaining over weakly linked metadata.
• Retail Banking: A high-fidelity synthetic lake derived from open-banking-inspired relational schemas (clients, accounts, transactions, policies), evaluating grounded decision-making under strict policy constraints.
• Biomedical Drug Information: A hybrid corpus fusing structured XML records from DrugBank 6.0 with open-access scientific literature from PubMed, stressing multimodal evidence integration.

The dataset contains 9,846 human-validated QA pairs. Each question is paired with a reference answer and exact, modality-aware evidence pointers (e.g., specific table rows or passage spans).

2. Construction Pipeline

The authors employ a reproducible five-stage pipeline to synthesize and validate the dataset:

1. Lake Instantiation: Preserving the domain's native schema.
2. Candidate Synthesis: Using a strong language model (GPT-5) to generate questions, answers, and evidence pointers from sampled subgraphs.
3. Structural Filtering: Programmatically removing malformed outputs.
4. Semantic Filtering: Using an LLM-as-a-judge to reject trivial candidates answerable via parametric memory alone.
5. Human Validation: Rigorous human review to verify Answerability (can the question be answered exclusively using cited evidence?) and Attribution (is the answer faithful to the cited text?).

3. Task Taxonomy

The benchmark organizes tasks into four reasoning families:

• Single-source lookups: Fact extraction from isolated objects.
• Cross-source composition: Combining facts across heterogeneous objects (e.g., joining a policy with a transaction).
• Multi-hop reasoning: Chained inference over intermediate entities or linked metadata.
• Anchor-based retrieval: Reasoning over related entities amidst semantically close distractors.

4. Evaluation Protocol

The evaluation isolates source discovery (retrieval) from cross-modal synthesis (generation). Metrics include:

• Accuracy: Binary correctness.
• Faithfulness (Attribution): Whether the predicted answer is supported by the predicted evidence set (AIS style).
• Retrieval Quality: Domain-aware recall of gold evidence.
• Token Usage: End-to-end token footprint.

Baselines evaluated include Zero-Shot prompting, standard Retrieval-Augmented Generation (RAG), and agentic tool-use methods (XMODE, RESP).

Key Results and Analysis

Baseline evaluations across the three domains reveal that high-quality retrieval does not guarantee correct reasoning, and failure modes are strongly domain-dependent.

• AI/ML Domain (Discovery Bottleneck): Retrieval quality is the primary bottleneck. While RAG achieves modest recall (e.g., 0.09 for Model questions), accuracy is higher (0.35), suggesting that data lakes contain factual redundancy. Systems can often synthesize correct answers from alternative valid evidence even if the specific "gold" artifact is missed. However, navigating weakly linked metadata for entity disambiguation remains difficult.
• Retail Banking Domain (Grounding Failure): The primary failure is not retrieval but grounding. Agentic systems (XMODE) achieve high policy recall (>0.85) but struggle to ground those abstract rules against specific operational ledgers to execute faithful decisions. Zero-shot models sometimes achieve high accuracy by relying on parametric priors for financial principles, but they fail the faithfulness metric (attribution), highlighting the risk of unauditable answers in enterprise settings.
• Drug Domain (Synthesis Bottleneck): This domain is characterized as a cross-modal synthesis challenge rather than a retrieval-limited one. While retrieval of passages and tables is often strong, end-to-end accuracy drops significantly when joint reasoning over structured and unstructured evidence is required (e.g., Passage-Table accuracy at 0.66 despite high individual recall). Multi-hop reasoning over structured biomedical evidence remains substantially harder than single-table lookups.

Ablation Study: When provided with perfect gold evidence (Context Bundle), AI/ML accuracy saturates at 98.7%, confirming it is a discovery problem. In contrast, Drug lake accuracy only reaches 62.5% with perfect evidence, indicating that cross-modal reasoning over dense biomedical tables is fundamentally broken in current LLMs, independent of retrieval performance.

Significance and Contributions

The paper claims three primary contributions:

1. The LAKEQUEST Benchmark: A human-validated dataset of over 9.8k questions evaluating grounded reasoning across three heterogeneous data lakes (AI/ML, Retail Banking, Biomedical Drug).
2. A Reproducible Construction Framework: A five-stage protocol for synthesizing, filtering, and validating QA pairs with explicit evidence traces over weakly structured corpora.
3. Empirical Analysis of Modern QA: Extensive baseline evaluations revealing that high retrieval quality does not consistently translate into accurate, faithful answers, especially when multi-step or cross-source composition is required.

The authors position LAKEQUEST as a necessary testbed to drive the next generation of reliable, agentic QA systems capable of rigorous discovery and faithful attribution over real-world data lakes. They emphasize that the benchmark is intended as a rigorous evaluation resource, not a substitute for domain expertise or human oversight in high-stakes decision pipelines.