💬 NLP

Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis

This paper proposes a cooperative framework that reframes ambiguity in natural language queries for tabular data as an intentional feature of user-system interaction, using this perspective to critique current evaluation practices and guide future research directions.

Original authors: Daniel Gomm, Cornelius Wolff, Madelon Hulsebos

Published 2026-03-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Gomm, Cornelius Wolff, Madelon Hulsebos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a tour guide (the computer system) and a tourist (the user) who wants to see the best sights in a massive, unfamiliar city (the database).

The paper "Are We Asking the Right Questions?" argues that we've been judging tour guides all wrong.

The Old Way: The "Perfect Tourist" Myth

Until now, researchers have treated the tourist like they should have a perfect, GPS-level map in their head. If the tourist says, "Show me the average summer temperature in Copenhagen," the old way of thinking says: "Great! You were specific. If you said 'Show me the temperature,' that's a failure because you didn't say WHERE or WHEN."

The paper says: Stop blaming the tourist.

In the real world, tourists don't know the city's layout. They rely on the guide to fill in the gaps. If a tourist says, "Show me the temperature," they aren't being lazy; they are trusting the guide to say, "Oh, you probably mean the current city we are in, and the last 30 days."

The New Idea: A Team Effort (Cooperative Interaction)

The authors suggest we view asking questions as a dance between two partners, not a test of one person's memory.

The "Cooperative" Query: The tourist says, "Show me the average summer temperature in Copenhagen."
- What's missing? They didn't say which years or exactly which months count as "summer."
- The Guide's Job: A good guide knows that "summer" usually means June-August and "average" usually means the mean. The guide fills in the blanks using common sense. This is a successful partnership.
The "Unambiguous" Query: The tourist says, "Show me the mean temperature in June-August in Copenhagen from 2000-2025."
- This is perfect, but it's rare. It's like the tourist bringing their own map and telling the guide exactly where to walk.
The "Uncooperative" Query: The tourist says, "Show me the temperature."
- This is a broken dance. The guide has no idea if the tourist means inside the building, outside, in Tokyo, or in 1990. The guide can't guess. This is a failure of communication, not just a lack of detail.

The Problem with Current Tests (The "Cheating" Datasets)

The paper looked at 15 different "training camps" (datasets) used to teach computers how to answer questions. They found two big problems:

The "Cheat Sheet" Problem (Data-Privilege):
Imagine a test where the tourist is allowed to peek at the guide's internal map before asking the question.
- Real Life: "Show me the top sales."
- Cheat Sheet: "Show me the top sales from the sales_2024.csv file, specifically the revenue column."
- The paper found that most tests let the "tourist" use the column names and file names. This isn't a real test of intelligence; it's a test of reading the cheat sheet.
The "Mixed Bag" Problem:
The tests mix up the three types of questions (Cooperative, Unambiguous, and Uncooperative) and grade them all the same way.
- If a guide gets a "Cooperative" question right by guessing the user's intent, the test might mark it wrong because it didn't match the "one true answer" in the answer key.
- If a guide gets an "Uncooperative" question right by guessing, the test might mark it wrong because the question was impossible to answer.
- Result: We don't know if the computer is smart at guessing (cooperation) or just good at following orders (execution).

The Solution: A New Rulebook

The authors propose we change how we build and test these systems:

Stop expecting perfection: Accept that users will leave things out. The system's job is to be a helpful partner who fills in the gaps using common sense, not a robot that demands perfect syntax.
Test the right skills:
- If you want to test if the computer can do math, give it a perfectly specific question.
- If you want to test if the computer is smart enough to understand humans, give it a vague but cooperative question and see if it makes a reasonable guess.
- If the question is too vague (Uncooperative), the computer should say, "I need more info," rather than guessing wildly.
No more cheat sheets: Future tests should hide the file names and column headers. The computer should have to figure out what data to look at just by listening to the human.

The Bottom Line

We need to stop treating natural language questions like code that must be perfect. Instead, we should treat them like a conversation between friends. A good AI shouldn't just be a calculator; it should be a collaborator that knows how to ask, "Did you mean this?" or "I assume you meant the latest data, right?"

By fixing how we ask questions and how we grade the answers, we can build systems that actually understand us, rather than just systems that follow instructions perfectly.

1. Problem Statement

The paper addresses a fundamental flaw in the current evaluation and design of Natural Language Interfaces (NLIs) for tabular data analysis.

The Ambiguity Misconception: Current systems and benchmarks treat ambiguity in user queries as a technical deficiency to be eliminated. The prevailing approach focuses on post-hoc detection and resolution to uncover a single "latent" user intent.
The Open-Domain Gap: In open-domain settings (unlike fixed-schema Text-to-SQL), users lack perfect knowledge of the underlying data structure. Consequently, they cannot provide "platonic" (perfectly specified) queries. They rely on implicit knowledge and shared conventions.
Evaluation Flaw: Existing benchmarks (e.g., Spider, BIRD, DA-Eval) mix different types of queries without distinction. They often contain "data-privileged" queries (where users reference specific column names or internal IDs they shouldn't know in an open setting) and conflate execution accuracy (getting the right answer for a clear query) with interpretation capability (making reasonable guesses for an ambiguous query). This makes it impossible to accurately diagnose system failures.

2. Methodology: The Cooperative Query Framework

The authors propose a paradigm shift: viewing ambiguity not as an error, but as a feature of cooperative interaction based on Grice's Cooperative Principle and the "least collaborative effort" heuristic.

A. Conceptual Framework

The framework categorizes queries based on the division of labor between the user (explicit specification) and the system (inference):

Unambiguous Queries: The user provides sufficient explicit information (or relies on universal conventions) such that the system has only one valid, actionable interpretation.
- Use Case: Evaluating execution accuracy against a gold standard.
Cooperative Ambiguous Queries: The user intentionally leaves details underspecified (e.g., "average summer temperature"), delegating the choice of specific parameters (e.g., defining "summer" or the specific correlation method) to the system. The system must use selective grounding to choose a reasonable option from a set of valid possibilities.
- Use Case: Evaluating a system's ability to make human-aligned, reasonable inferences.
Uncooperative Queries: The query is underspecified to the point of irresolvable ambiguity (e.g., "What is the average temperature?" with no location or context).
- Use Case: Evaluating system robustness and the ability to request clarification.

B. Grounding Mechanisms

The paper defines two mechanisms for deriving "actionable interpretations":

User-Provided Grounding: Explicit data or context in the query.
System-Inferred Grounding:
- Conventional Grounding: Resolving ambiguity via universal common sense (e.g., "highest mountain" $\to$ "in the world").
- Selective Grounding: The system chooses from a set of reasonable options when the user delegates the decision (e.g., choosing Pearson vs. Spearman correlation).

C. Empirical Analysis

The authors analyzed 15 diverse datasets spanning Tabular QA, Text-to-SQL, and Data Analysis (e.g., Spider, BIRD, DA-Eval, KramaBench).

Classification Tool: They developed LLM-based classifiers (validated against human experts) to assess two dimensions:
1. Data-Independence: Detecting "data-privileged" queries (structural references, value references, container references).
2. Cooperative Interpretability: Classifying queries as Unambiguous, Cooperative (requiring selective grounding), or Uncooperative.

3. Key Contributions

Theoretical Reframing: Proposes that ambiguity is an intentional signal of user intent and a mechanism for division of labor, rather than a system failure.
Evaluation Framework: Introduces a principled method to separate execution accuracy (tested on unambiguous queries) from interpretation capability (tested on cooperative ambiguous queries).
Benchmark Audit: Provides the first systematic analysis of 15 major benchmarks, revealing that:
- Many complex analysis benchmarks are saturated with data-privileged queries (users referencing specific column headers or internal IDs), violating the open-domain premise.
- Unambiguous queries are a minority in complex analysis datasets (e.g., DA-Eval, DA-Code), meaning current benchmarks cannot effectively test pure execution accuracy.
- Existing evaluations conflate different capabilities, leading to misleading performance metrics.
Design Guidelines: Offers concrete directions for future system design, including:
- Moving from single-shot execution to mixed-initiative dialog for uncooperative queries.
- Designing systems that disclose their grounding choices (selective grounding) to allow user intervention.
- Creating datasets with multiple valid grounding paths to test iterative query refinement.

4. Results

Data-Privilege: A significant portion of queries in complex analysis benchmarks (like DA-Eval and DA-Code) rely on knowledge inaccessible to a real open-domain user (e.g., referencing "column X" or specific internal IDs).
Query Type Distribution:
- Unambiguous queries are rare in complex analytical tasks (often <20-30% in analysis-focused datasets).
- Most queries in current benchmarks are either procedurally specified but data-underspecified (requiring selective grounding) or completely uncooperative.
Evaluation Mismatch: Current benchmarks fail to isolate system capabilities. A system might fail because it cannot interpret a vague query (an interpretation failure) but is penalized as if it failed to execute a clear query (an execution failure).

5. Significance and Future Directions

For Research: The paper argues that the field must stop treating ambiguity as a bug. Future research should focus on selective grounding—teaching systems to make human-aligned choices when users are vague, rather than forcing systems to guess a single "correct" answer that may not exist.
For Evaluation: Benchmarks must be stratified. Researchers should report separate scores for:
- Execution accuracy (on unambiguous queries).
- Interpretation quality (on cooperative queries).
- Robustness (on uncooperative queries).
For System Design: NLIs should evolve from "query executors" to "cooperative partners." Systems should be designed to:
- Recognize when a query requires selective grounding.
- Present multiple reasonable interpretations to the user.
- Engage in mixed-initiative dialogue to resolve irresolvable ambiguities rather than hallucinating an answer.

In summary, the paper provides a critical lens for the Tabular Data Analysis community, arguing that the path to robust, open-domain AI requires redefining the relationship between user intent and system inference, moving away from rigid "correctness" toward flexible, cooperative interpretation.