🤖 machine learning

TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

This paper introduces TraceML, a comprehensive dataset and schema pairing human and agent trajectories across 134 machine learning competitions, which reveals that current agents fail to match human performance not due to coding errors but because they lack the strategic planning and iterative exploration behaviors exhibited by human experts.

Original authors: Jiarui Yan, Weiwei Sun, Sijie Li, Wenhan Li, Yiming Yang

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiarui Yan, Weiwei Sun, Sijie Li, Wenhan Li, Yiming Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a distinct difference between writing a single correct sentence and building a complex machine that learns from data. Large language models, the powerful computer programs that can draft emails or write code for isolated tasks, have become remarkably skilled at the former. However, when asked to act as autonomous engineers—loading messy data, choosing the right mathematical models, training them, and then spending hours tweaking their approach based on what the results tell them—they often stumble. They can produce a final answer, but they struggle to navigate the long, winding road of discovery that leads to it. This gap is not just about getting the wrong number at the end; it is about how the work is done. While human experts might try a dozen different strategies, discard the ones that fail, and return to old ideas that seemed promising, these automated agents often get stuck in narrow loops, repeating the same small adjustments without ever truly changing direction.

Researchers at Carnegie Mellon University set out to understand exactly where and why this breakdown happens. They created a new way of looking at the work of both human experts and artificial agents, treating their entire development process as a single, continuous story rather than just a final score. They gathered thousands of records of how humans solved machine learning competitions, capturing every time they saved a new version of their code, every time they changed a model, and every time they adjusted their data. They then paired this with the work of two different types of automated agents working on the same problems. By lining up these histories side by side, they could see not just who won, but how each side thought, moved, and reacted to failure over the course of a competition.

What they found was a clear divide in how the two groups approached the problem. The human experts moved with a fluid, adaptable rhythm. They would spend time cleaning data, then switch to testing a new model, then step back to verify their results, and sometimes, when they hit a wall, they would go back to an idea they had abandoned days earlier. They treated the process as a series of experiments, willing to pivot completely when the evidence suggested a new path was needed. The automated agents, by contrast, behaved like they were trapped in a single room. One type of agent, which relied on a standard command-line interface, spent almost all its time making tiny, repetitive adjustments to a single solution it had already found, like constantly re-weighing the same set of ingredients without ever trying a new recipe. The other type of agent, which used a search method to try many variations at once, changed its model constantly but failed to consolidate those changes into a coherent strategy. It would jump from one idea to another so frequently that it never gained any real ground, and it almost never returned to a previous approach to see if it could be improved.

The researchers discovered that the agents were not just slower; they were missing a crucial human capability: the ability to remember and revisit. When a human expert realized a current path was a dead end, they could look back at their history, find a promising dead end from earlier in the day, and try to fix it. The agents, however, seemed to have no memory of their own history. They could recover from a bad score by tweaking the current model, but they could not reopen a line of work they had set aside. This lack of memory meant they were constantly reinventing the wheel or spinning their tires, unable to build on the lessons of their own past attempts. The study showed that while the agents could recover from setbacks faster than humans in some cases, they lacked the strategic patience to abandon a failing strategy and return to a better one later.

To test if this behavior was simply a matter of giving the agents better instructions, the researchers tried a new approach. They created a set of guidelines based on how the human experts actually worked, instructing the agents to switch between different types of tasks, to stop tweaking the same thing over and over, and to try new models more often. When they gave these instructions to the agents, the behavior did change, but only in specific ways. The agents started to follow the rules about when to switch tasks and how to build a team of models, and their scores improved on several competitions. However, the underlying rhythm of their work remained unchanged. They still did not return to old ideas, and they still did not possess the deep, flexible memory that allowed humans to navigate the long hours of development. The instructions could tell the agents what to do, but they could not teach them how to think about their own history.

The study concludes that the gap between human and artificial intelligence in this field is not just about raw computing power or the ability to write code. It is about the architecture of the search itself. The agents are missing the ability to treat their own past work as a resource to be mined, rather than just a list of steps to be executed. While a simple set of instructions can fix some of the surface-level behaviors, the deeper issue requires a fundamental change in how these systems are designed. They need a way to remember their journey, to recognize when they are stuck in a loop, and to have the confidence to go back and try a different path. Until they can do that, they will remain powerful tools that can follow a map, but they will struggle to be the explorers who draw the map in the first place.

Technical Summary: TraceML

Problem Statement

While Large Language Models (LLMs) demonstrate proficiency in writing correct code for isolated, well-specified tasks, they remain significantly less capable in autonomous machine-learning (ML) development. In this domain, an agent must iteratively load data, clean pipelines, select and train models, interpret validation signals, and revise strategies over extended periods (hours). Despite steady progress within individual runs, agents consistently finish below strong human competitors in Kaggle-style competitions and derive less benefit from additional working time.

Current outcome-based benchmarks (e.g., MLE-bench, MLAgentBench) record this performance gap but fail to explain its cause. By grading only the final submission, they discard the development process, rendering two runs with identical scores indistinguishable even if one involved careful experimentation and the other blind tuning. The paper posits that the divergence between agent and expert workflows is a question of process, requiring data that captures the sequence of edits, intents, and state transitions rather than just the final outcome.

Methodology: TraceML

The authors introduce TraceML, a dataset and schema designed to pair human and agent work on the same competitions under a unified, version-level representation.

1. Data Construction

Human Corpus: Reconstructed from the Meta Kaggle database and Code mirror, covering 4,465 human trajectories across 134 competitions. The pipeline involves:
- Ingestion: Extracting public saved versions, deduplicating via hashing, and joining with author tiers and leaderboard scores.
- Lineage Reconstruction: Recovering development order via a directed acyclic graph (DAG) using notebook histories, fork relationships, and code similarity.
- Pruning: Filtering out post-deadline edits, shallow lineages, and "score-fishing" resubmissions (score changes without code changes).
- Normalization: Converting all data into a version-level format.
Agent Corpus: Paired with the human data on 7 specific competitions. It includes:
- Codex: 18 runs total (11 baseline runs and 7 runs carrying the planning prompt).
- MLEvolve: 189 runs (branches from a search tree) using an evolutionary search strategy.
- Total: 207 agent trajectories.
- Note on Analysis Scope: While 18 Codex runs were collected, the behavioral comparisons in §4 are computed on a twelve-hour scope subset containing 10 Codex runs (Table 6), which share a matched budget with the human trajectories.
Alignment: Both human and agent runs are mapped to a common schema where a "version" is a deliberate save (human) or a submission-producing commit/search node (agent). Both are scored using the competition's own metric (normalized to within-competition percentiles).

2. Annotation Schema

TraceML annotates every code version and transition with a competition-agnostic vocabulary:

Version State: Assigns each version to one of 8 coarse ML pipeline stages (e.g., feature engineering, ensembling) with 136 fine-grained tags.
Transitions: Labels the change between versions across four dimensions:
- Action: Operations performed (fine-grained).
- Intent: The purpose of the edit.
- Magnitude: The size of the code rewrite.
- Score Effect: Whether the metric improved, plateaued, or regressed.

3. Labeling Pipeline

Given the scale (151,088 versions), manual labeling is infeasible. The authors distill the schema into open-weight labelers:

A teacher model (gpt-5.4-mini) generates schema-constrained labels on curated traces.
Two student models (Qwen3-1.7B) are trained for State Labeling and Action/Intent Labeling.
Reliability is verified via schema compliance, cross-model agreement, and human audits, showing high consistency (e.g., Cohen's $\kappa \approx 0.87$ for state, $\approx 0.61$ for intent).

Key Empirical Findings

The analysis of paired trajectories reveals consistent behavioral differences between humans and agents:

1. Action Profiles: Narrow Loops vs. Broad Mixing

Humans: Alternate between exploration, diagnosis, validation, model changes, and ensembling. They frequently pivot and revisit abandoned lines.
Codex: Collapses into a narrow loop focused on submission-facing bookkeeping. It spends most steps re-weighting existing ensembles, stacking models, and tweaking post-processing. It rarely changes direction (9% pivot rate vs. 25% for humans).
MLEvolve: Focuses on local model mutations (editing layers, changing epochs, averaging seeds). It pivots frequently (58% rate) but without consolidating gains; its net gain per pivot is near zero.
Gap: Neither agent scaffold engages in "changing direction" (e.g., swapping checkpoints, changing pretrained sources) or reopening abandoned work at human rates.

2. Memory and Recovery

Recovery: Agents can recover from setbacks (Codex recovers faster than top humans), but they do not revisit abandoned approaches.
Revisits: Top humans return to earlier non-adjacent versions 9% of the time (78% of those returns improve the score). Codex revisits once in the entire dataset; MLEvolve never does. Agents lack a searchable history of their own work.

3. Ensembling

Human Practice: Ensembling involves adding new members and blending diverse models.
Agent Practice:
- Codex: Re-weights existing members without adding new ones.
- MLEvolve: Averages seeds of the same model.
- Result: While all cohorts "ensemble," only human ensembling correlates with score improvement. Agent ensembling is "in name only" regarding its impact on performance.

Intervention: The Planning Harness

The authors constructed a planning prompt (the "harness") distilled from human diagnostics to test if instruction can close the gap.

Mechanism: A ~1,000 token prompt enforcing anti-loop constraints (e.g., no single hold-out, no repeated hyperparameter tweaks) and prescribing human-prior practices (K-fold from the start, early ensembling, OOF caching).
Results:
- Behavioral Shift: The prompt successfully moved three behaviors toward human profiles: reduced re-weighting, increased attention to ensembling, and increased small early edits.
- Performance: Scores improved in 5 of 7 competitions.
- Limitations: The prompt failed to change the rhythm of development (agents still did not switch modes as frequently as humans) and overshot prohibitions (e.g., eliminating hold-outs entirely, whereas humans still use them 25% of the time).
- Conclusion: Instruction can close the gap for behaviors that reduce to "yes/no" checks but cannot fix the underlying design problems of memory and control.

Contributions

Dataset (TraceML): A version-level trajectory dataset of 4,465 human and 207 agent runs across 134 competitions, released with extraction code and labeler checkpoints.
Schema: A unified extraction and annotation schema for Kaggle notebooks, CLI commits, and tree-search journals, enabling side-by-side comparison.
Analysis: Process-level evidence of human-agent gaps in exploration, validation, model switching, and memory.
Use Case: A planning harness that demonstrates the boundary between what instruction can fix and what requires agent architecture changes.

Significance and Claims

The paper claims that TraceML transforms the abstract observation "agents score lower" into specific, actionable design problems.

Memory: Agents lack a form of history they can search to revisit abandoned work, a capability distinct from simple score recovery.
Control: Agents follow a single policy throughout a run. The gap is not in the base model's capability but in the controller that reads the run's state and decides when to pivot.
Instruction vs. Design: The study concludes that instruction (prompting) can only address the part of the gap that reduces to explicit rules. The remaining gap—where agents fail to pivot or revisit—lies in agent design (architecture and memory mechanisms), not just prompting.

The authors release the corpus, schema, and pipeline to serve as a reference distribution for future agents, allowing new systems to be evaluated against human practice as they emerge, rather than relying on static benchmarks that age with the field.