Technical Summary: DataMaster: Towards Autonomous Data Engineering for Machine Learning

1. Problem Formulation

The paper addresses a shift in the Machine Learning (ML) landscape where improvements in model architectures, training recipes, and compute resources are becoming increasingly standardized. Consequently, performance gains are increasingly dependent on data quality, selection, and composition. However, current autonomous ML systems remain largely model-centric, treating data as a fixed input or an auxiliary action dimension rather than the primary object of optimization.

The authors formulate task-conditioned autonomous data engineering as a new research problem. In this setting:

• Input: A downstream task $T$, a fixed learning algorithm $a_0$, an initial executable data state $D_0$, and a resource budget $B$.
• Constraint: The learning algorithm $a_0$ remains unchanged.
• Objective: An autonomous agent must maximize downstream performance $E(T, a_0, D)$ by operating exclusively on the data state $D$.
• Challenges: The problem is open-world (useful data sources are scattered across the web with unknown quality and licensing) and long-horizon (the value of a data decision is only verifiable after downstream training and evaluation). The agent must discover, clean, adapt, and integrate data while avoiding repeated failures and leveraging cumulative knowledge.

2. Methodology: The DataMaster Framework

DataMaster is a memory-augmented data-agent framework designed to navigate the open-ended search space of data engineering. It unifies external data discovery, executable data refinement, and cumulative memory under a downstream training feedback loop. The framework consists of three core components:

A. DataTree (Structured Search)

The DataTree organizes the search over executable data states as a tree structure $T = (V, E)$, where nodes represent data-engineering steps and edges represent context inheritance. It employs two complementary node types:

• Red Nodes (Data Exploration): These nodes perform open-ended external search to discover new datasets. They generate search queries based on task descriptions and failure cases, screen sources for relevance and usability, and output a set of new dataset entries $\Delta P_v$ to be added to a shared Data Pool.
• Black Nodes (Data Exploitation): These nodes leverage the Data Pool to improve the current training pipeline. They select sources, apply cleaning/transformation rules, construct features, and adapt inputs to the fixed algorithm. They materialize these decisions into an executable DataLoader and submit it for downstream training to obtain a performance score $y_v$.

B. Shared Data Pool

This component serves as a global repository for candidate external datasets discovered by Red nodes. It stores manifests containing source pointers, schema summaries, and screening notes. Black nodes read from this pool to select and compose data, ensuring that discoveries made in one branch are reusable by others, preventing redundant search efforts.

C. Global Memory

Global Memory acts as a persistent record layer for the search process. It stores node-level records $r_v$ containing:

• Outcomes: Downstream scores and diagnostics.
• Artifacts: Reusable processing histories and cleaning methods.
• Findings: Useful hints and failure modes.
  This allows the agent to retrieve context from parent and sibling nodes, enabling cross-branch learning (e.g., avoiding failed transformations or reusing successful adapters).

D. Tree Growth and Scheduling

To manage limited computational budgets, DataMaster employs:

• Growth Policy: A dynamic mechanism where a Red node is followed by a batch of Black nodes to test the utility of newly discovered data. The system decides whether to deepen a branch (further refinement) or broaden it (new discovery) based on training feedback.
• Scheduling Policy: A greedy scheduler based on an Upper Confidence Bound (UCB) rule. It selects the next node to execute by balancing the cumulative reward of a branch against the need to explore under-visited directions, ensuring efficient allocation of the evaluation budget.

3. Key Contributions

1. Problem Formulation: The paper defines task-conditioned autonomous data engineering as a first-class problem where the data state, not model code, is the primary optimization target.
2. Framework Design: It introduces DataMaster, a novel framework integrating a DataTree for branching search, a Data Pool for shared external sources, and Global Memory for cumulative reuse across iterations.
3. Empirical Validation: The authors evaluate DataMaster on two complementary benchmarks, demonstrating that structured data-side iteration can significantly improve strong initial solutions in both classical ML engineering and LLM post-training.

4. Experimental Results

The authors evaluated DataMaster on MLE-Bench Lite (22 Kaggle competitions) and PostTrainBench (7 LLM post-training capabilities).

• MLE-Bench Lite:

  • DataMaster improved the Medal Rate by 32.27% over the initial score (reaching 68.18% vs. 35.91%).
  • It achieved a Gold Medal Rate of 45.45%, surpassing the strongest baseline (DatasetResearch) by 9.09 percentage points in medal rate.
  • Ablation studies confirmed that removing Global Memory caused a collapse in performance (Gold Medal Rate dropped from 80% to 20%), highlighting the necessity of cumulative memory.

• PostTrainBench:

  • DataMaster raised the average score from 8.47% (base model) to 31.17%.
  • It outperformed all baselines, including Codex (18.46%) and DataFlex (19.09%).
  • Notably, on the GPQA benchmark (graduate-level scientific QA), DataMaster surpassed the human-expert instruct model (31.02% vs. 30.35%).

• Robustness and Scaling:

  • The system showed consistent gains across different backbone LLMs (GLM-5, Claude-4.6, Qwen3.5), though GPT-5.4 underperformed due to insufficient tool usage.
  • Test-time scaling experiments showed that increasing the search budget led to continuous performance improvements, with the final node surpassing the instruct model reference.

5. Significance and Claims

The paper claims that autonomous data engineering is a viable and powerful lever for improving ML systems, distinct from and complementary to model-centric optimization.

• Paradigm Shift: The work argues that as model development becomes standardized, the "main remaining lever" for performance gains is data. Therefore, agentic systems must move beyond optimizing model code to optimizing the data state.
• Structured Search: The paper demonstrates that treating data engineering as a structured, tree-based search process with cumulative memory is superior to isolated data operations or simple dataset retrieval.
• Generalizability: The results suggest that this approach is effective across diverse domains, from tabular ML competitions to complex LLM post-training, and can even match or exceed expert-curated data pipelines in specific capabilities.

The authors conclude that future autonomous ML systems should place greater emphasis on the full lifecycle of data: discovery, selection, transformation, validation, and reuse.