💻 computer science

DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

This paper introduces DataClawEval, the first comprehensive benchmark for evaluating autonomous agents in real-world, end-to-end data engineering tasks across five execution engines, revealing that current frontier models lack general proficiency and remain limited by strict domain specialization.

Original authors: Debin Meng, Jiaming Yang, Zefang Zong, Tengyue Xu, Haining Xie, Yang Li, Peng Chen

Published 2026-07-31

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Debin Meng, Jiaming Yang, Zefang Zong, Tengyue Xu, Haining Xie, Yang Li, Peng Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just chat with you but actually do things. This is the realm of AI Agents. Think of them not as smart chatbots that give you advice, but as digital interns that can open a laptop, read a messy spreadsheet, figure out what needs to be fixed, write the code to fix it, and then check their own work to make sure it actually works. For a long time, scientists have tested these digital interns on simple tasks, like translating a sentence into a database query (a bit like asking a librarian to find a book based on a vague description). But the real world of data is much messier. It involves connecting different types of databases, handling live streams of information, and debugging code that crashes in unexpected ways. The big question researchers are asking is: Can these AI agents actually handle the complex, real-world job of a professional data engineer, or do they just look good on paper?

Enter DataClawEval, a new "stress test" for these AI agents, created by researchers at Tencent and Xidian University. Instead of asking the AI to just write a single line of code, this benchmark throws them into a realistic industrial simulation. The researchers built a sandbox containing 100 different data engineering tasks, ranging from analyzing user growth to managing security risks. These tasks require the AI to use five different "engines" (specialized tools like PySpark, MySQL, and FlinkSQL) to build, run, and debug entire data pipelines. The twist? The AI isn't just graded on whether it writes the right words; it's graded on whether the code actually runs and produces the correct data in a live environment.

The results of this experiment were a bit of a reality check. The researchers tested 16 of the smartest AI models available today. Even the "champion" model, GPT 5.5, only managed to get a score of 74.9 out of 100. This suggests that while AI is getting better, the dream of a fully autonomous data engineer is still far from solved. The study found that no single AI model is a master of all trades; some are great at one type of database but terrible at another. For instance, while most models handled MySQL tasks well, they struggled significantly with HiveSQL. Furthermore, the researchers discovered that using more "brain power" (consuming more computer tokens) didn't necessarily lead to better results. In fact, the most efficient agents were often the ones that didn't just guess and retry endlessly.

Perhaps the most surprising finding was about how we should grade these AI agents. The team tested whether a second AI could act as a judge to score the work, but it turned out to be a disaster. The "AI judge" was overly generous, giving high scores to agents that failed to run their code correctly, simply because the text looked nice. The researchers concluded that only a strict, rule-based system that actually runs the code and checks the data can tell the truth. In short, DataClawEval shows us that while AI agents are promising, they still have a lot of growing up to do before they can be trusted to run our critical data systems without a human looking over their shoulder.

Technical Summary: DataClawEval

Problem Statement

While Large Language Models (LLMs) and autonomous agents have shown promise in software engineering and Text-to-SQL tasks, the domain of end-to-end enterprise data engineering remains largely unexplored in existing benchmarks. Current evaluation efforts typically isolate single capabilities: Text-to-SQL benchmarks (e.g., BIRD, Spider 2.0) focus on query generation over fixed schemas; code-execution benchmarks (e.g., InfiAgent-DABench) measure analytical answers rather than deployable pipelines; and exploratory analysis benchmarks lack the rigor of production-grade engineering workflows.

Real-world data engineering differs fundamentally from these tasks. It requires agents to inspect database schemas, understand business semantics, reason about transformation logic, implement executable programs across heterogeneous engines (batch and streaming), iteratively debug runtime failures, and validate production outputs. Existing benchmarks fail to capture this complexity, lacking realistic ETL tasks, multi-engine support, reproducible runtime environments, and deterministic protocols that verify the entire engineering process rather than just a final answer.

Methodology

Dataset Construction: DataClawEval

The authors introduce DataClawEval, the first executable benchmark designed specifically for autonomous data engineering agents. The dataset comprises 100 rigorous, end-to-end tasks derived from production-grade code authored by professional enterprise data engineers at Tencent.

Scope: The tasks span five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. They cover both offline (batch) and online (streaming) workloads.
Domains: Tasks are drawn from five business domains: Ops & Resource Governance, Data Analytics & User Growth, Security & Risk Control, Content & Community, and Advertising & Marketing.
Construction Pipeline: The authors employ a human-in-the-loop methodology to ensure answer-identifiable tasks:
1. Curation: High-quality production code is desensitized, deduplicated, and stratified.
2. Reconstruction: An LLM infers user intent and synthesizes input tables based on the ground-truth code.
3. Iterative Validation: A differential testing loop ensures synthesized inputs uniquely characterize the ground-truth implementation. Human experts perturb the code to verify that incorrect implementations produce distinguishable results.
4. Grading Scripts: Case-specific, deterministic grading scripts are developed by experts to evaluate schema correctness, numerical accuracy, and business logic.
5. Final Audit: Domain experts verify the consistency of intent, inputs, code, and grading criteria.

Evaluation Framework

DataClawEval operates within isolated, case-specific Docker containers to prevent information leakage and ensure reproducibility.

Artifact-Oriented Evaluation: Instead of comparing source code, the framework evaluates the correctness of produced data artifacts (tables/files) against business-specific criteria.
Process-Oriented Evaluation: The framework tracks the agent's execution trajectory, measuring exploration adequacy, execution efficiency, and self-verification behaviors.
Scoring: The final score is a weighted combination ( $Score = \alpha S_{artifact} + (1-\alpha) S_{process}$ ), with $\alpha=0.7$ in most cases, prioritizing final correctness while rewarding efficient engineering processes.
Deterministic Grading: The authors explicitly reject "LLM-as-a-Judge" for scoring, citing systematic score inflation, non-determinism, and bias. They utilize rule-based scripts that execute outputs against live databases.

Key Results

The authors evaluated 16 frontier LLM-based agents (including GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, DeepSeek V4, and others) using a unified agent harness (Tencent CodeBuddy).

Significant Headroom: The field is far from saturated. The strongest model, GPT-5.5, achieved an overall score of only 74.9/100, indicating substantial room for improvement.
No Dominant Model: No single model dominates across all engines. Performance is highly engine-specific:
- MySQL is the easiest (scores >74 for all agents).
- HiveSQL is the hardest (no agent exceeds 69.8; some fall below 60).
- PySpark and FlinkSQL show the largest performance spreads, effectively separating capable agents from weaker ones.
Token Cost vs. Quality: There is no positive correlation between token consumption and solution quality. Some models (e.g., Gemini 3.5 Flash) consume significantly more tokens with mid-tier scores, while others (e.g., GPT 5.5) achieve top-tier accuracy with lower token budgets.
Stability Issues: Repeated runs reveal that high peak performance does not guarantee robustness. Models like Hy3 showed large score gaps between best and worst runs, and pass^3 (solving in all three runs) was significantly lower than pass@3 (solving in any run) for several agents.
Tool Call Efficiency: Tool call frequency is uncorrelated with success. Leading agents achieve high scores with fewer tool calls, while inefficient agents accumulate calls without improving artifacts.

Significance and Claims

The paper claims that DataClawEval establishes a new standard for evaluating autonomous agents in realistic industrial data engineering scenarios. Its primary contributions are:

First Executable Benchmark: It is the first benchmark grounded in real-world commercial data sources that requires agents to complete end-to-end workflows (schema inspection, coding, debugging, and materialization) rather than just generating code snippets.
Deterministic, Rule-Based Evaluation: By rejecting LLM judges in favor of case-specific, executable grading scripts, the benchmark provides a trustworthy, fine-grained, and reproducible assessment of agent capabilities.
Revealing Domain Specialization: The evaluation exposes that current agents possess strict domain specialization rather than omnipotent proficiency, with performance heavily dependent on the specific execution engine.
Reliability Gaps: The results highlight that autonomous data engineering remains a formidable, unresolved challenge, with current models lacking the stability and efficiency required for unattended production workloads.

The authors release the dataset, containerized environments, and deterministic evaluation scripts to facilitate further research in this critical domain.