🤖 AI

Can Generalist Agents Automate Data Curation?

This paper introduces Curation-Bench to demonstrate that while generalist coding agents can automate the data curation loop and match existing baselines, achieving superior, research-level data policies requires scaffolding that forces agents to cite and adapt prior methods rather than relying on open-ended prompting.

Original authors: Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced student (an AI model) how to solve complex puzzles. You have a massive library of 665,000 practice problems, but you only have time to give them 10,000. The question is: How do you pick the best 10,000?

In the past, humans had to do this manually. They would guess which problems were good, test the student, see where they failed, and then tweak their list. It was slow, expensive, and exhausting.

This paper asks: Can a smart computer agent (a "Generalist Agent") do this job for us? Can it act like a research assistant that picks the data, trains the model, checks the results, and tries again automatically?

Here is the story of what they found, using some simple analogies.

1. The Setup: The "Cooking Contest"

The researchers built a special kitchen called CURATION-BENCH.

The Chef (The Agent): A smart AI that can read files, write code, and run commands.
The Ingredients (The Data): A huge pile of mixed-up recipes (images and text).
The Rules: The agent cannot change the oven temperature (the training code) or the judging panel (the tests). The only thing the agent can change is which ingredients it puts in the pot.
The Goal: Create a small batch of 10,000 ingredients that makes the dish taste as good as a dish made with the full 665,000.

2. The First Attempt: "The Intuitive Cook"

The researchers let the agents cook with open-ended prompts. They basically said, "Go ahead, pick the best 10,000 recipes you can find."

The Result: The agents were surprisingly good at the basics.

They quickly figured out that just grabbing random recipes was bad.
They started tweaking the mix: "Let's add more cooking recipes and fewer math problems," or "Let's make sure we have enough long, detailed instructions."
The Analogy: Think of this like a home cook who knows the basics. They don't need a textbook to know that if the soup tastes too salty, they should add more water. The agents could "tune" the recipe by adjusting simple knobs (like the ratio of different data sources).
The Limit: They got stuck in a rut. They kept tweaking the same recipe over and over (e.g., "Maybe 60% cooking, 40% math? No, maybe 55/45?"). They rarely thought to try a completely different cooking style, like "What if we rewrite the recipes instead of just picking them?"

3. The Problem: "The Execution Gap"

The agents were great executors (they could run the loop and follow instructions) but poor researchers (they didn't know how to explore new ideas).

Even when the researchers gave them a list of "cool cooking techniques" or a stack of "famous cookbooks" (scientific papers), the agents mostly ignored them. They would say, "I'll try diversity sampling!" in their notes, but then they would actually just change the ratio of ingredients again. They were stuck in "local optimization"—tweaking the same small area instead of exploring the whole kitchen.

4. The Solution: "The Strict Sous-Chef" (Scaffolding)

To fix this, the researchers introduced Scaffolds. Think of these as strict rules or training wheels that force the agent to think like a real scientist.

They tried two types:

Light Scaffolds: "Hey, here are some cool techniques you could try." (This made the agents talk about more ideas, but they still cooked the same way).
Heavy Scaffolds: "Before you change the recipe, you must cite a specific cookbook, explain exactly why you are using that technique, and show how you adapted it to your ingredients."

The Breakthrough:
The Heavy Scaffolds worked like magic.

The agents stopped just tweaking ratios. They started reading the "cookbooks" (scientific papers) and actually implementing complex, new strategies.
One agent discovered a method called EL2N (which is like picking the recipes the student struggled with the most, but then filtering out the ones that were just broken or confusing).
The Result: This "scaffolded" agent created a dataset of 10,000 examples that performed better than human-designed baselines that used 100,000 examples. They achieved the same (or better) results with one-tenth of the data.

5. The Big Takeaway

The paper concludes that:

Agents can run the loop: They are great at doing the repetitive work of testing and tweaking.
But they need a guide: Without strict rules forcing them to cite evidence and adapt specific methods, they get stuck in a "vibe check" mode, making small, safe changes rather than big discoveries.
Compute is the new data: If you can't get more data, you can spend more "computer time" (iterations) searching for the perfect way to use the data you already have.

In short: You can give a smart AI a job to curate data, but if you want it to be a true researcher and not just a tweaker, you have to give it a strict checklist that forces it to learn from past discoveries rather than just guessing.

Technical Summary: Can Generalist Agents Automate Data Curation?

Problem Statement

Modern AI development relies heavily on training data curation, a process that is both consequential and labor-intensive. Practitioners typically engage in an iterative loop of proposing, implementing, evaluating, and revising data policies based on noisy benchmark feedback. While prior work has automated individual data operations (e.g., pruning, deduplication, rewriting), the higher-level decision-making process—deciding which technique to use, how to adapt it, and when to revise it—remains largely manual. Existing benchmarks either fix the data and optimize training recipes (e.g., ML-research agents) or treat data as a static input for analysis (e.g., data-science agents), leaving a gap in evaluating iterative data-policy discovery by autonomous agents.

Methodology: CURATION-BENCH

To address this gap, the authors introduce CURATION-BENCH, an agent-centric benchmark designed to evaluate whether generalist coding agents can automate the data-curation loop.

Benchmark Design Principles

Data Isolation: The model architecture, training recipe, and evaluation suite are fixed. The agent's only variable is the curated data policy.
Terminal Realism: Agents operate in a standard terminal workspace, interacting with files, writing scripts, running commands, and debugging, mirroring real-world engineering workflows.
Contamination Control: An automatic audit checks submitted datasets for exact matches or high-overlap text spans against evaluation benchmarks to prevent data leakage.
Trajectory Legibility: The system records the full curation trajectory (scripts, manifests, logs, scores) to evaluate not just the final outcome, but the quality of the search process.

Task Formalization

A task is defined by a tuple $\tau = (M_0, D_{pool}, B, T, E, C)$ , where $M_0$ is the initial model, $D_{pool}$ is the candidate pool, $B$ is the data budget, $T$ is the fixed training procedure, $E$ is the evaluation suite, and $C$ represents constraints. The agent must produce a policy $\pi$ that maps $D_{pool}$ to a curated dataset $D_\pi$ ( $|D_\pi| \le B$ ). The policy is scored by $S(\pi) = E(T(M_0, D_\pi))$ .

Experimental Setup

The primary instantiation focuses on multimodal instruction tuning:

Data: Selecting 10k samples from the LLaVA-665K dataset (665k total) to fine-tune the LLaVA-1.5-7B model.
Agents: Generalist coding agents (Claude Code, Codex, Kimi, Qwen) operating via the OpenHands harness.
Scaffolds: The study compares open-prompt agents against light scaffolds (optional guidance on strategies or papers) and heavy scaffolds (binding protocols requiring evidence-grounded hypotheses or mandatory adaptation of prior methods).

Key Contributions

1. Benchmark for Iterative Data Curation

CURATION-BENCH is the first benchmark to isolate training-data curation as the central object of optimization for autonomous agents, recording the full search trajectory to distinguish between effective exploration and local tuning.

2. The Execution–Research Gap

The study identifies a persistent gap where agents are capable executors but limited researchers.

Execution: Open-prompt agents can reliably run the curation loop, producing valid datasets and improving training outcomes over random selection.
Research Limitation: Agents tend to "over-exploit" local policy families (e.g., adjusting source ratios, length thresholds, or random seeds) rather than exploring distinct policy families. Even with access to strategy guides and paper references, agents often fail to operationalize abstract research ideas into executable policies without further structural constraints.

3. The Role of Scaffolding

The paper demonstrates that scaffolding is the critical lever for converting agent execution into genuine research exploration.

Light Scaffolds: Providing lists of strategies or paper references broadens the agent's stated options but rarely changes the executed policy, as agents default to low-cost local heuristics.
Heavy Scaffolds: Imposing binding protocols (e.g., "Adapt Papers") that require agents to cite a specific prior method, explain its applicability, and adapt it to the current budget forces a shift toward method-guided exploration. This approach enables agents to discover novel, high-performing policies that open-prompt agents miss.

Results

Performance of Open-Prompt Agents

Baseline Achievement: Open-prompt agents reach strong published data-selection baselines (e.g., ICONS, ARDS) within ten iterations.
Efficiency: They recover approximately 59% of the full-data fine-tuning gain using only 1.5% of the original data pool (10k vs. 665k).
Limitation: Their success is driven by local heuristic tuning rather than exploring new policy families.

Impact of Heavy Scaffolding

Breakthrough Performance: Under the "Adapt Papers" scaffold, an agent autonomously composed a hybrid policy combining EL2N-style top-loss selection with an assistant-loss noise filter.
Superiority: This scaffolded agent achieved an average score of 34.9, outperforming both the open-prompt agent and strong non-agent baselines (ICONS, ARDS) while using only one-tenth of the data budget required by those baselines (10k vs. 100k).
Trajectory Quality: Heavy scaffolds eliminated "shallow" operations (dropping from 47% to 0% in the best case) and increased "grounded" iterations to 100%.

Scaling Compute Budget

Extending the agent's iteration budget from 10 to 50 shows that performance continues to improve without a clear plateau, suggesting that in finite-data regimes, additional compute can be effectively spent on searching over data policies rather than just training on more data.

Significance and Claims

The paper claims that current generalist agents can run the curation loop but cannot reliably conduct data research without scaffolded method adaptation.

Autonomous vs. Assisted: The distinction is drawn between autonomous execution (running a loop with clear local verification) and agent-assisted research (navigating a discontinuous policy space with sparse feedback). Data curation falls into the latter category.
Scaffolds as Artifacts: The authors argue that scaffolds are not merely prompts but experimental design objects that determine the agent's reasoning capabilities. Future evaluations of autonomous data research must report scaffold strength and trajectory diagnostics, not just final scores.
Data-Efficient Compute: The work suggests a new paradigm for improving model performance in finite-data regimes: spending extra compute on searching over data policies (selection, adaptation, validation) rather than solely on reusing data or synthetic expansion.

The authors conclude that while open-ended prompting yields useful executors, reliable data research requires scaffolded method adaptation to bridge the gap between abstract methodological knowledge and executable data policies.