🤖 AI

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

This paper introduces Ambig-DS, a benchmark comprising two diagnostic suites that reveal how data-science agents silently fail by committing to unintended task framings when faced with ambiguity, highlighting that recognizing underspecification rather than ensuring pipeline execution is the critical bottleneck in current evaluations.

Original authors: Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens, Robert Kitchen

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens, Robert Kitchen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Silent Mistake"

Imagine you hire a very smart, automated chef (an AI agent) to cook a meal for you. You give them a recipe card and a basket of ingredients.

In most current tests, the recipe card is perfect: it says, "Make a spicy pasta dish using tomatoes and basil." The chef cooks it, serves it, and the test says, "Great job! The pasta is cooked and plated correctly."

But in the real world, recipe cards are often messy. You might just say, "Make something with these ingredients," without specifying what to make.

The Problem: The AI chef sees the ingredients and silently decides, "Okay, I'll make a salad." They make a perfect, edible salad. But you wanted pasta.
The Failure: The test says, "Success! The salad is cooked and plated." The test doesn't know you wanted pasta. The AI didn't ask, "Did you want pasta or salad?" It just guessed, made a perfect wrong dish, and hid the mistake behind a perfectly executed task.

The paper calls this "Unflagged Misframing." The AI is technically competent (it can cook), but it lacks the wisdom to realize the instructions were vague and to ask for clarification before starting.

The Solution: Ambig-DS (The "Confusion Test")

The researchers created a new benchmark called Ambig-DS to catch this specific type of failure. Instead of giving the AI perfect instructions, they intentionally created "tricky" tasks where the goal is unclear.

They tested two main types of confusion:

1. The "Which Number?" Confusion (Target Ambiguity)

The Setup: Imagine a spreadsheet with two columns of numbers. One column is the "real" answer the AI should predict (e.g., "Total Sales"), and the other is a "decoy" (e.g., "Total Employees"). Both look very similar and are equally easy to predict.
The Trick: The instructions just say, "Predict the value." They don't say which value.
The Result: The AI picks one (usually the decoy), builds a perfect model, and submits it.
The Score: Because the AI picked the wrong column, it gets a terrible score, even though the code ran perfectly.
The Finding: Even the smartest AI models (the "frontier" models) fell for this trap 39% to 63% of the time. They silently committed to the wrong answer.

2. The "How Do We Measure?" Confusion (Objective Ambiguity)

The Setup: Imagine a task where you have to guess if a photo has a cactus in it. The instructions say, "Submit your guesses," but they forget to say how the guesses will be graded.
The Trick: Should you submit a "Yes/No" (Hard Label) or a "70% chance of Yes" (Probability)?
- If the grader wants probabilities, but you submit Yes/No, you fail.
- If the grader wants Yes/No, but you submit probabilities, you might fail or get a weird score.
The Result: The AI guesses. Sometimes it guesses the wrong format. Sometimes, realizing it doesn't know, it just gives up and submits a lazy, constant answer (like "Yes" for everything) just to get the task done.
The Finding: In these cases, 16% to 62% of the time, the AI either guessed the wrong format or gave up silently.

The "Magic Question" Experiment

The researchers wanted to know: If the AI was allowed to ask one question, could it fix the mistake?

They gave the AI a "Clarification Oracle" (a magic button that answers one question truthfully).

The Result: When the AI was allowed to ask, "Which column is the target?" or "Do you want probabilities or Yes/No?", their performance shot back up to near-perfect levels.
The Catch: The AI is great at answering the question if it asks. But the AI is terrible at knowing when to ask.
- If you tell the AI "You can ask anything," it asks too many silly questions (like "Do you like spicy food?") on tasks that were already clear.
- If you tell the AI "Only ask if you are stuck," it stays silent on the tricky tasks and makes the wrong guess anyway.

The Takeaway

The paper concludes that we are currently testing AI agents like we test race cars: we see if they can drive fast and stop at the right time. But we aren't testing if they can read a map when the road signs are missing.

The main bottleneck isn't that the AI can't write code or train models; it's that the AI doesn't know when it doesn't know.

For Users: Don't assume the AI understands your vague instructions. Be very specific about what you want to predict and how you want to measure success.
For Builders: We need to train AI to be better at saying, "I'm confused, can you clarify?" rather than just guessing and hoping for the best.
For Testers: We need to stop just checking if the code runs. We need to check if the AI realized the instructions were vague in the first place.

In short: An AI that can perfectly execute a wrong plan is a dangerous AI. Ambig-DS is the first tool designed to catch that specific kind of silent failure.

Technical Summary: Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

Problem Statement

As data-science agents evolve from co-pilots to auto-pilots, a critical failure mode emerges: silent misframing. Unlike execution errors (e.g., code crashes or syntax errors), silent misframing occurs when an agent commits to a plausible but unintended interpretation of a task's goal. The agent produces a clean, executable pipeline and a valid submission artifact, yet the underlying task framing (prediction target or evaluation objective) is incorrect. Because the pipeline runs successfully and the submission format is correct, standard benchmarks—which primarily score pipeline execution and artifact validity—fail to detect this failure.

Current benchmarks (e.g., DS-1000, DSBench, MLE-bench) typically provide fully specified prompts where the prediction target, output form, and evaluation metric are explicitly defined. However, real-world data science workflows often lack these commitments; stakeholders provide a goal and data, leaving the modeler to decide what constitutes the target and success. The paper argues that the ability to recognize when a task is underspecified (i.e., when the prompt-data observation does not justify a unique commitment) and to ask for clarification before acting is a missing competence in current agent evaluations.

Methodology

The authors introduce Ambig-DS, a paired diagnostic benchmark designed to isolate and measure task-framing ambiguity. The benchmark consists of two independently analyzed suites:

Ambig-DS-Target (51 tasks): Built on DSBench (tabular modeling).
- Intervention: Controlled prompt-data edits where the true target column is renamed (e.g., to val_1) and a plausible decoy target (val_2) is added. The prompt is rewritten to use generic language (e.g., "predict value") rather than specific target names.
- Construction: The decoy target is engineered to have the same marginal distribution as the true target but is driven by a different set of features. Noise is calibrated so that the decoy is nearly as predictable from features as the true target, preventing agents from distinguishing them via simple correlation or learnability checks.
Ambig-DS-Objective (61 tasks): Built on MLE-bench (Kaggle-style competitions).
- Intervention: Prompt-only edits where explicit metric references (e.g., "ROC-AUC", "RMSE") are removed and replaced with generic phrases like "a standard evaluation procedure." The data, target, and submission format remain unchanged.
- Goal: To create ambiguity regarding whether the agent should submit probabilities, hard labels, or how to optimize (e.g., maximize vs. minimize).

Verification Protocol:
Each ambiguous variant undergoes a rigorous verification pipeline involving human review and cross-family LLM audits against a four-item checklist:

Plausible Alternatives: At least two reasonable framings exist.
Ambiguity Preservation: No cues leak the intended framing in the prompt or data.
Decision Relevance: Resolving the ambiguity changes a critical task-level choice (e.g., target selection, output type).
Task Preservation: The edit does not alter unrelated task information.

Evaluation Conditions:
Agents are evaluated under three conditions:

Full: The original, fully specified task.
Ambig.: The ambiguous variant with no clarification channel. Agents must commit silently.
Ask: The ambiguous variant where the agent may ask one self-contained clarification question (≤50 words) to a Clarification Oracle before modeling. The oracle answers truthfully but is restricted to the specific missing framing variable (target name or metric details) and refuses out-of-scope requests.

Key Contributions

Formalization of Task Framing: The paper isolates two core framing variables: the prediction target ( $T$ ) and the evaluation objective ( $M$ ). It defines "unflagged misframing" as the failure to recognize underspecification and commit to an unintended framing without asking.
Ambig-DS Benchmark: The release of two diagnostic suites (112 paired tasks total) constructed via controlled interventions that preserve the original source-benchmark evaluators. This ensures performance drops are attributable solely to framing ambiguity, not scoring rule changes.
Empirical Quantification: The paper quantifies the prevalence of silent misframing and the calibration of "ask-before-act" behavior across five frontier and efficient code-based agents (Gemini 3 Flash/Pro, GPT-5.4 Nano/Pro, Claude Haiku 4.5).

Results

1. Silent Misframing is Prevalent

Across all agents, ambiguity induces significant performance degradation:

Target Ambiguity: Agents commit to a wrong target in 39–63% of ambiguous runs (depending on the model). These are valid, executable pipelines that solve the wrong problem.
Objective Ambiguity: Agents exhibit silent failure in 16–62% of runs. This splits into two modes:
- Misframing: Optimizing for a plausible but unintended metric (e.g., accuracy instead of AUC).
- Abdication: Submitting a degenerate baseline (e.g., constant predictions) rather than attempting to resolve the ambiguity.
Silent Commitment: Even when agents acknowledge ambiguity in their internal traces, they rarely flag the final commitment as user-overridable. The failure is characterized by a "silent commitment" to a default framing.

2. Clarification Recovers Performance

When agents are allowed to ask a single question to the oracle:

Recovery: Performance recovers significantly, with $\Delta_{ask}$ (Ask score minus Ambig score) ranging from +0.03 to +0.20.
Capacity vs. Detection: Agents demonstrate the capacity to integrate framing information when provided perfectly (recovery is high), but they lack the calibration to detect when to ask.
Calibration Failure: No model-prompt combination achieves robust calibration:
- Permissive Prompts: Lead to high recall on ambiguous tasks but high false positives (asking unnecessary questions) on fully specified tasks.
- Conservative Prompts: Reduce false positives but suppress legitimate asking on ambiguous tasks, leading to silent defaults.

3. Failure Signatures

Target Suite: Failures are discrete wrong-target submissions.
Objective Suite: Failures are a mix of substantive misframing (wrong metric) and abdication (giving up).
Cross-Suite Consistency: The pattern of degradation under ambiguity and recovery under clarification holds across different model families (Gemini, GPT, Claude) and is not an artifact of the specific oracle or judge models used.

Significance and Claims

The paper claims that recognizing target and objective underspecification is the bottleneck missing from standard data-science agent evaluations.

Decoupling Competence: The work separates pipeline-execution competence (can the agent run code?) from task-framing competence (does the agent know what to solve?). Standard benchmarks conflate these by assuming the task is fully specified.
Actionable Signals:
- For Users: Framing variables (target, metric, output type) must be specified explicitly; omitting them causes reliable performance drops even for capable agents.
- For Builders: "Ask-act calibration" should be a concrete target for post-training or prompt engineering. Current prompt strategies shift the operating point but do not yield robust sensitivity to decision-relevant ambiguity.
- For Benchmark Designers: Benchmarks should pair fully specified tasks with diagnostic ambiguous variants to avoid overestimating agent competence.

The authors maintain a modest stance, noting that Ambig-DS is a controlled diagnostic suite, not a census of all real-world ambiguity. The results estimate failure modes under verified, decision-relevant ambiguity rather than providing a prevalence rate for all data-science tasks. The study highlights that while agents can use missing information if offered, they currently cannot reliably detect when they need it.