💬 NLP

SANA: What Matters for QA Agents over Massive Data Lakes?

This paper introduces SANA, a diagnostic ablation framework that decomposes end-to-end Exploratory Question Answering (EQA) performance over massive data lakes into specific component failures—search, planning, and data analysis—to systematically identify bottlenecks and guide improvements in agent design.

Original authors: Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a very smart, but sometimes confused, detective (an AI Agent) to solve a mystery. The clues aren't in one neat file; they are scattered across a massive, chaotic warehouse filled with millions of boxes, papers, and digital files (a Data Lake).

The detective's job is to find the right clues, read them, do some math, and solve the case. But sometimes, the detective fails. The big question is: Why? Did they look in the wrong place? Did they misunderstand the clue? Did they do the math wrong? Or did they just get confused about what to do next?

This paper introduces SANA, a new way to diagnose exactly where the detective goes wrong. Think of SANA as a "black box" recorder that lets you pause the investigation and swap out specific parts of the detective's brain to see which one is the weak link.

The Three Main Ways a Detective Can Fail

The authors break down the detective's job into three main skills:

Searching (Finding the Clues): Can the detective find the right boxes in the massive warehouse?
- The Problem: If the warehouse is huge, the detective might grab the wrong boxes or miss the right ones entirely.
Planning (The Game Plan): Can the detective figure out the right order of steps? (e.g., "First find the location, then find the people who live there, then count them.")
- The Problem: The detective might get confused, skip a step, or try to solve the puzzle backwards.
Data Analysis (Doing the Work): Once the clues are found, can the detective actually read them and do the math?
- The Problem: The detective might find the right document but misread the numbers or write the wrong code to calculate the answer.

How SANA Works: The "Super-Powered" Swap

To figure out which skill is the problem, SANA uses a clever trick. It takes a solved mystery (where we already know the answer) and creates a "Gold Standard" version of the clues.

Then, it runs the detective through the same mystery but swaps out their tools one by one:

The "Perfect Search" Swap: We give the detective a magic wand that only hands them the correct boxes, skipping the need to search. If the detective still fails, the problem isn't searching; it's something else.
The "Perfect Plan" Swap: We give the detective a pre-written map of the exact steps to take. If they still fail, the problem isn't planning; they just aren't following the map.
The "Perfect Math" Swap: We give the detective a calculator that guarantees the right answer if they ask the right question. If they still fail, the problem is that they are asking the wrong questions.

By comparing the detective's normal performance against these "perfect" versions, SANA can pinpoint exactly where the breakdown happens.

What They Found: The Detective's Weak Spots

The researchers tested this on two different types of "mysteries" (benchmarks): LakeQA (a huge, messy warehouse) and KramaBench (a smaller, more focused set of clues).

Here is what they discovered:

1. The "Data Analysis" Bottleneck (The Math Problem)
In both types of mysteries, the biggest problem was Data Analysis. Even when the detective found the right clues and had a good plan, they often messed up the actual calculation or code. It's like having the right ingredients but burning the cake. This was the most consistent failure point.

2. The "Search" Bottleneck (The Warehouse Problem)
In the LakeQA (huge warehouse) setting, Searching was a huge problem. The detective got lost in the massive amount of data.

The Fix: When they gave the weaker detective a "magic wand" that only showed the right boxes, their performance jumped up massively. This proves that for huge data lakes, the ability to find the needle in the haystack is the hardest part.
The Contrast: In the smaller KramaBench setting, searching wasn't as big of a deal because there were fewer boxes to look through.

3. The "Planning" Bottleneck (The Map Problem)
Surprisingly, Planning wasn't the biggest issue. The detectives were actually pretty good at coming up with a decent plan on their own.

The Catch: The problem wasn't making the plan; it was following it. Even when given a perfect map, the weaker detectives often got distracted, took a wrong turn, or forgot where they were going. They struggled to stick to the path.

The "Residual" Problem: The Human Error

Even when the researchers gave the detective the Perfect Search, the Perfect Plan, and the Perfect Math tools, they still didn't get 100% of the answers right.

Why? The detective's "Action Policy" (their decision-making brain) still had flaws. They would sometimes:
- Give up too early.
- Submit the wrong answer even though they had the right clues.
- Get stuck in a loop doing the same thing over and over.

The Bottom Line

This paper doesn't just say "AI is getting better." It acts like a mechanic's diagnostic tool. It tells us that:

If you are working with massive data, your AI needs better search skills.
If you are doing complex data analysis, your AI needs better coding/math skills.
Regardless of the task, your AI needs better discipline to stick to its plan and not give up or hallucinate the final answer.

SANA helps developers stop guessing and start fixing the specific part of the AI that is actually broken.

Technical Summary: SANA - Search Agent Navigation Ablation Framework

1. Problem Definition

Exploratory Question Answering (EQA) over data lakes requires an LLM agent to iteratively discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results. Unlike standard QA, the agent must determine which tables, metadata, filters, or computations are necessary to answer a high-level question without prior knowledge of the data schema.

Current evaluation methods rely on end-to-end accuracy, which conflates failures across distinct components:

Search: Retrieving irrelevant, incomplete, or missing sources.
Planning: Decomposing the question into wrong or incomplete subquestions.
Data Analysis: Writing incorrect queries/code, misinterpreting results, or schema errors.
Agent Policy: The turn-level decision process (Action Policy) determining what to do next, when to stop, or how to synthesize evidence.

Prior benchmarks (e.g., LakeQA, KramaBench) and agentic evaluations (e.g., DCI, Metadata Reasoner) do not isolate these failure modes. Consequently, it remains unclear whether an agent fails due to poor retrieval infrastructure, weak reasoning chains, or flawed execution policies.

2. Methodology: The SANA Framework

The authors propose SANA (Search Agent Navigation Ablation framework), a diagnostic ablation framework designed to deconstruct end-to-end EQA task accuracies into specific component failures.

2.1 Task Profiling

SANA transforms solved EQA tasks into runtime profiles containing ground-truth information:

Source Sequence: The ordered sequence of gold datasets ( $D_{gold}$ ) required for each subquestion.
Sanitized Subquestions: A sequence of subquestions ( $\tilde{Q}$ ) where direct mentions of dataset names are removed to prevent leaking search targets (e.g., replacing "nypd_crimes" with "find crime reports reported by NYPD").
Execution Records: Specifications of the dataset, the analysis intent, the correct code/query, and the intermediate answer for each step requiring data analysis.

2.2 Intent-Based Ablation

SANA isolates the contribution of system components by replacing them with idealized versions while preserving the agent's Action Policy (decisions on what to do next). The framework defines three axes for ablation:

Planning Ablation: The agent receives the sanitized subquestion sequence ( $\tilde{Q}$ ) and the goal of each step but must still formulate search queries and execute analysis. This tests if the agent can follow a correct decomposition.
Search Ablation: The standard search tool is replaced with an oracle-like operator ( $f_{ideal}$ ) that restricts the search space strictly to $D_{gold}$ . The agent still decides what to search for and when, but the retrieval precision is perfect.
Data Analysis Ablation: The standard execution tool is replaced with $g_{ideal}$ , which checks if the agent's intent matches the annotated execution record. If matched, it returns the verified intermediate answer; otherwise, it uses a stronger model to generate correct code for the stated intent.

2.3 Evaluation Setup

The framework was applied to two benchmarks:

LakeQA: A large-scale data lake setting (~40 million documents, 135 tasks).
KramaBench (Converted): A smaller-scale, data-to-insight setting (1,764 sources, 83 tasks) adapted to require data discovery.

Experiments were conducted with two model sizes (gpt-5.4-nano and gpt-5-mini) under fixed budgets ( $B=30$ tool calls) and runtimes. The study compares Naive (baseline), Standard (current best practices), and Ideal (ablation) configurations.

3. Key Results

3.1 LakeQA Findings (Large-Scale Discovery)

Search is a Major Bottleneck: Ideal search (oracle retrieval) improved semantic match (SM) by +14.1% for gpt-5.4-nano and +13.3% for gpt-5-mini over BM25. Preloading gold sources yielded even higher gains (+28.9% for nano), indicating that weaker agents struggle significantly with search navigation (formulating queries and committing to sources).
Data Analysis is a Consistent Bottleneck: Ideal data analysis improved SM by +18.5% for gpt-5-mini, suggesting stronger models often identify the correct intent but fail in implementation (SQL/Python generation).
Planning is Less Limiting: Ideal planning provided modest gains (+3.0% for nano, +9.6% for mini). Analysis revealed that agents can often generate plans similar to the gold decomposition, but they fail to follow them during execution (trajectory drift).
Residual Policy Failures: Even with all components idealized, a gap remains (e.g., 76.3% SM for gpt-5-mini vs. 100% potential). Failures include hallucinating final answers, wasting budget on loops, or failing to validate intermediate evidence.

3.2 KramaBench Findings (Smaller-Scale Analysis)

Data Analysis Dominates: In this setting, data analysis was the primary bottleneck, improving SM by +24.1% (nano) and +14.5% (mini) when idealized.
Search is Less Critical: Search improvements were marginal (+2.4% to +6.0%), reflecting the smaller scale and the nature of the tasks which are closer to data-to-insight pipelines than broad discovery.
Planning Impact: Planning ablation showed more significant relative gains for the weaker model (nano) than the stronger one, suggesting explicit decomposition helps weaker agents more in structured analysis tasks.

3.3 Failure Analysis

Audit of failed traces revealed distinct failure profiles:

gpt-5.4-nano: Failures were distributed across source targeting, plan divergence, turn-waste (repeated loops), and premature stopping.
gpt-5-mini: Failures were concentrated in execution/computation (57.2% of failures) and finalization (35.7%). This model typically reached the correct evidence but failed to implement the operation correctly or synthesize the final answer.

4. Contributions

SANA Framework: A diagnostic ablation framework that converts solved EQA tasks into runtime profiles to construct idealized interfaces for planning, search, and data analysis while preserving the agent's policy.
Controlled Evaluation: A systematic evaluation on LakeQA and converted KramaBench that isolates the end-to-end contributions of search, planning, and execution under fixed conditions.
Empirical Diagnosis: Evidence that data analysis is a consistent limitation across benchmarks, search is a major limitation in large-scale discovery, and residual policy failures (trajectory control, validation, stopping criteria) persist even when components are idealized.

5. Significance and Claims

The paper claims that SANA provides a necessary shift from evaluating "black-box" end-to-end accuracy to a diagnostic approach that identifies specific failure modes in EQA agents.

Deconstruction of Accuracy: The authors argue that end-to-end accuracy alone cannot distinguish between a failure in search, planning, or the agent's policy. SANA allows for systematic comparisons of progress in these specific areas.
Policy vs. Component Quality: The results suggest that future improvements in EQA agents require more than just better retrievers or stronger code generation. The "residual gap" indicates a need for stronger runtime policies that can track subgoal progress, commit to validated sources, recover from trajectory drifts, and verify intermediate evidence before submitting a final answer.
Context-Dependent Bottlenecks: The paper emphasizes that bottlenecks are not universal; they shift based on the benchmark structure (e.g., search dominates in large lakes, while analysis dominates in smaller, insight-driven tasks).

Limitations Acknowledged by Authors:

Diagnostic Approximation: Ideal tools are approximations of oracles; they may not credit alternative valid sources outside the profile.
Task Scope: Conclusions are specific to the evaluated task suites (LakeQA and KramaBench) and may not generalize to all agent benchmarks.
Judge Reliance: Several metrics (semantic match, plan trajectory, failure labeling) rely on LLM judges, which may introduce false positives/negatives.