KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes

The paper introduces KramaBench, a comprehensive benchmark featuring 104 real-world data-to-insight challenges across diverse domains, which reveals that current AI systems struggle to orchestrate end-to-end data pipelines over data lakes, achieving a maximum of only 55% accuracy despite strong performance in isolated tasks.

The Gist

Imagine you are a detective trying to solve a complex mystery, but instead of a single notebook of clues, you have been handed a massive, chaotic warehouse filled with thousands of boxes. Some boxes are neatly labeled, some are full of scribbles on napkins, some are broken, and some are empty. Your goal is to find a specific piece of information hidden inside this mess and present it as a clear answer.

This is exactly what KRAMABENCH is about. It is a new "test" designed to see if Artificial Intelligence (AI) can act like a real data detective.

Here is a breakdown of the paper in simple terms:

1. The Problem: The "Messy Warehouse"

In the real world, data scientists don't just get clean spreadsheets. They get "Data Lakes"—huge collections of messy, unorganized files from different sources (like old government records, sensor logs, or medical reports).

• The Challenge: To get an answer, an AI has to do a lot of work: find the right files, clean up the garbage, combine different pieces of information, and do the math.
• The Gap: While AI is great at writing code or answering simple questions, nobody knew if it could handle this whole messy process from start to finish.

2. The Solution: KRAMABENCH (The "Obstacle Course")

The researchers built a giant obstacle course called KRAMABENCH.

• The Course: It contains 104 real-world puzzles based on 1,700 actual files from 6 different fields (like archaeology, astronomy, and law).
• The Rules: The AI is given a question (e.g., "How much money was lost to fraud in 2024?") and the entire messy warehouse. It must figure out which files to open, how to clean them, and how to calculate the answer—all by itself.
• The Twist: To make sure the AI isn't just "cheating" by memorizing answers from its training, the researchers changed some of the numbers and names in 20% of the puzzles. If the AI relies on memory, it fails. If it actually reasons through the data, it succeeds.

3. The Test: Putting AI to the Work

The researchers tested 8 different AI systems (including big names like GPT-4o, Claude, and OpenAI's "Deep Research") on this course. They also built their own simple helper tool called DS-Guru to see how a basic AI performs.

The Results: The AI is still a "Junior Intern"
The results were a bit of a reality check:

• The Best Score: Even the smartest AI system only got 55% of the puzzles right from start to finish.
• The "Perfect" Score: Even when the researchers gave the AI the exact right files (removing the need to search), the score only went up to 62%.
• The Bottleneck: The AI is good at the "big picture" (it can guess the general plan) but terrible at the "details" (it often messes up the specific math or code needed to execute the plan).

4. Why Did the AI Struggle? (The "Why")

The paper found three main reasons why the AI failed:

1. The "Needle in a Haystack" Problem: The AI often couldn't find the right files in the huge warehouse. It would get distracted by irrelevant data.
2. The "Fine-Grained" Glitch: The AI could write a plan, but when it tried to execute a small step (like fixing a typo in a date format), it would break the whole chain. It's like a chef who can write a great recipe but burns the toast every time they try to make it.
3. The "Mind Reading" Failure: Sometimes the data was ambiguous (e.g., a beach name that looked like a street name). A human expert would use common sense to figure it out. The AI, however, would get stuck or guess wrong because it lacked "common sense" or prior knowledge.

5. The Human Comparison

The researchers also asked 9 human data scientists to solve the same puzzles.

• Humans did better (about 76% accuracy), but they still made mistakes!
• The Lesson: Even humans struggle with messy data. The biggest mistake humans made wasn't coding errors; it was designing the wrong plan. This proves that the hardest part of data science isn't typing code; it's figuring out how to solve the problem.

The Big Takeaway

KRAMABENCH shows us that while AI is getting smarter, it isn't ready to replace a human data scientist yet.

• Current AI: It's like a very fast, very confident intern who can read a lot of books and write a draft plan, but needs a human to double-check the math, find the right files, and fix the mistakes.
• Future Goal: We need to build AI that doesn't just "guess" the answer but can actually navigate the messy warehouse, clean the data, and build a working machine to solve the problem on its own.

In short: AI can write the recipe, but it's still learning how to cook the meal without burning the kitchen down.

Technical Summary

Here is a detailed technical summary of the paper "KRAMABENCH: A BENCHMARK FOR AI SYSTEMS ON DATA-TO-INSIGHT PIPELINES OVER DATA LAKES".

1. Problem Statement

The paper addresses the critical gap in evaluating Large Language Model (LLM) agents on end-to-end data science workflows. While current benchmarks excel at isolated tasks (e.g., code generation, text-to-SQL, or single-step reasoning), they fail to capture the complexity of real-world data science, which involves:

• Heterogeneous Data: Processing unclean, semi-structured, and unstructured data from "data lakes" containing tens to thousands of files.
• Multi-Step Orchestration: The need to discover relevant data, clean/normalize it, integrate multiple sources, and perform statistical analysis to derive insights.
• Domain Knowledge: Solving problems that require specific domain context (e.g., legal, biomedical, astronomy) and handling implicit conventions.
• Robustness: Dealing with noisy inputs, missing values, and schema inconsistencies without human intervention.

Existing benchmarks often rely on curated, clean inputs or fine-grained prompts, which do not reflect the "data-to-insight" challenge where the system must autonomously navigate a raw data lake.

2. Methodology: KRAMABENCH

The authors introduce KRAMABENCH, a comprehensive benchmark designed to test the full lifecycle of data science automation.

Dataset Composition

• Scale: 104 manually curated tasks spanning 1,700+ files from 24 data sources across 6 domains: Archaeology, Astronomy, Biomedical Research, Environmental Science, Legal Insight Discovery, and Wildfire Prevention.
• Task Design: Each task is a natural language query requiring the system to ingest the entire domain data lake, identify relevant files, clean data, and execute a pipeline to produce a specific answer (numeric, string, or list).
• Ground Truth: Every task includes:
  1. A reference end-to-end solution (code).
  2. Key Functionalities: A list of essential logical steps any correct pipeline must contain.
  3. Sub-tasks: Granular prompts for individual steps (e.g., "Which column contains temperature?") with ground truth answers.
• Data Leakage Control: To prevent models from relying on training data memorization, 20% of tasks use obscured inputs (synthetic identifiers and numeric values) while maintaining the original task structure.

Evaluation Framework

KRAMABENCH evaluates systems across three distinct settings to diagnose failure modes:

1. End-to-End Automation: The system must solve the task without human intervention. Success is measured by the accuracy of the final output against the ground truth.
2. Pipeline Design: The system is evaluated on its ability to identify the necessary Key Functionalities (logical steps) required to solve the problem, even if the code execution fails.
3. Sub-task Implementation: The system is given a specific sub-task prompt and evaluated on its ability to implement that single step correctly.

Reference System: DS-Guru

To provide a baseline, the authors created DS-Guru, a lightweight scaffolding framework that guides an LLM through the process:

• Retrieval: Uses One-Pass Sampling (OPS) to provide schema summaries and row samples from the data lake (avoiding context window limits).
• Reasoning: Employs Chain-of-Thought prompting to decompose tasks.
• Iteration: Uses a few-shot approach where the LLM sees execution errors and attempts to repair the pipeline.

3. Key Contributions

1. First End-to-End Data Lake Benchmark: KRAMABENCH is the first benchmark to evaluate AI systems on complex, multi-file, multi-step data science pipelines over raw, uncurated data lakes.
2. Granular Evaluation Framework: It moves beyond "pass/fail" on the final answer by separately measuring pipeline design capability (reasoning) and sub-task implementation (coding/execution), allowing for precise diagnosis of where systems fail.
3. Comprehensive Baseline Analysis: The paper evaluates 8 LLMs across 4 different agent architectures (including single-agent, multi-agent reflexion, and closed-source "Deep Research" tools), providing the first comparative analysis of these approaches on data-intensive workflows.
4. Insight into Failure Modes: The study identifies that current systems struggle not just with retrieval, but with fine-grained data-dependent reasoning and holistic understanding of data conventions.

4. Experimental Results

The authors evaluated 8 LLMs (including GPT-o3, GPT-4o, Claude-3.7, Llama-3.3, DeepSeek-R1) using DS-Guru and various agent frameworks (smolagents, OpenAI Deep Research, Gemini Agentic).

Key Findings

• Low End-to-End Success: Even the best-performing system (smolagents DR with Claude-3.7) achieved only 55.83% accuracy in the full data lake setting.
• Retrieval is Not the Primary Bottleneck: Providing "Oracle" inputs (only the correct files, skipping discovery) only improved accuracy by 6–7% on average. This suggests the main failure point is reasoning and pipeline design, not just finding the right files.
• The "Implementation Gap":
  • Systems can identify ~42% of necessary pipeline steps (Pipeline Design score).
  • However, they can only fully implement ~20% of individual sub-tasks correctly.
  • This indicates a disconnect between high-level reasoning and low-level code execution on noisy data.
• Agent Architectures:
  • Agentic Control Flow Wins: Systems with iterative "search-plan-repair" loops (smolagents DR) significantly outperformed structured, single-shot frameworks (DS-Guru).
  • Multi-Agent vs. Single-Agent: A multi-agent Reflexion setup (Actor-Evaluator-Reflector) performed similarly to the single-agent DR baseline (55.37% vs 55.83%), suggesting simple evaluator loops do not yet solve the complexity of data workflows.
• Knowledge Leakage: When inputs were obscured (synthetic data), performance dropped drastically (e.g., from 62.81% to 12.77% for Reflexion), indicating that current models heavily rely on parametric knowledge (memorized facts) rather than genuine data-dependent reasoning.
• Human Baseline: Human experts achieved 76.75% accuracy, but nearly 46% of their errors were due to incorrect pipeline design, highlighting that even humans struggle with the cognitive load of multi-file data integration.

5. Significance and Future Directions

• Realistic Assessment: KRAMABENCH demonstrates that current "Deep Research" and agentic systems are not yet ready for fully autonomous data science. They can generate plausible drafts but fail to produce working, robust pipelines over real-world messy data.
• Research Directions: The paper identifies specific areas for improvement:
  • Data-Dependent Reasoning: Models need to better infer data semantics (e.g., recognizing "M" as missing values) from execution errors.
  • Holistic Understanding: Agents need to integrate prior domain knowledge to decipher ambiguous data conventions (e.g., encoding schemes in file names).
  • Clarification: Systems should be designed to ask clarifying questions when data is underspecified, rather than hallucinating assumptions.
• Community Resource: The benchmark, including code, data, and evaluation scripts, is open-sourced to accelerate research in automated data science.

In conclusion, KRAMABENCH establishes that while LLMs are powerful code generators, the leap to autonomous data scientists capable of navigating unstructured data lakes requires significant advances in iterative reasoning, error recovery, and domain-aware data interpretation.