Technical Summary: HASTE – Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering

Problem Statement

Current Machine Learning Engineering (MLE) agents, as evaluated on benchmarks like MLE-Bench, operate largely as "cold starts." Despite the existence of reusable techniques across competitions, agents reset their state between tasks, forcing them to rediscover known solutions from scratch. This redundancy consumes significant compute budgets and relies heavily on frontier models or extended time limits to compensate for repeated exploration. While recent work has explored cross-task transfer, existing systems often store knowledge in flat pools or vector stores. This "flat" organization dilutes the signal within the context window, making it difficult for agents to select the right prior for a specific task, leading to inefficient exploration and suboptimal performance.

Methodology: HASTE Architecture

The authors propose HASTE (Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering), a multi-agent system designed to organize accumulated knowledge by scope and load it hierarchically.

1. Hierarchical Skill Store

The core innovation is a plain-text skill store organized into three scope tiers, stored as markdown files with YAML frontmatter:

• Global Tier: Skills applicable across all ML tasks (e.g., "Ensembling a strong model with a weaker one can degrade performance").
• Domain Tier: Skills specific to modalities (Tabular, Vision, NLP, Audio).
• Competition Tier: Skills tied to specific datasets or unique quirks of a single competition.

The store contains three types of entries, each serving a distinct function in the agent's loop:

• Technique Entries: Record what worked or failed (e.g., "Target encoding helps high-cardinality categoricals").
• Commitment Priors: Identify design choices with high cross-task variance, signaling that evidence is required before committing to a specific architecture.
• Refinement Hints: Specify which hyperparameters or "knobs" to tune for specific model families.

2. Agent Hierarchy and Execution

The system employs a two-level hierarchy:

• Orchestrator: Classifies competitions into domains, schedules execution, and manages skill promotion. After each round, an LLM evaluates new learnings to decide whether to skip them, keep them at the competition level, abstract them to the domain level, or promote them to the global level. Conflicts are resolved by annotating conditions (e.g., "Ensembling helps if correlation < 0.95").
• Domain Specialists: Separate agents for Tabular, Vision, and NLP tasks. Each specialist executes a five-stage pipeline:
  1. Task Profiling: Analyzes metadata and resources.
  2. Prototype Screen: Proposes and tests three diverse model approaches.
  3. Adaptive Refinement: A linear refinement loop (Exploring → Optimizing → Fine-tuning) with auto-escalation and revert-on-regression mechanisms.
  4. Ensemble: Combines top checkpoints.
  5. Learning Production: Reflects on the experiment history to generate new skills.

3. Scoped Context Loading

Crucially, agents do not load the entire skill inventory. The orchestrator loads only the skills matching the agent's scope (Global + relevant Domain + Competition-specific). This keeps the context focused, preventing signal dilution and ensuring the agent sees only relevant priors.

Key Contributions

1. 3-Tier Hierarchical Skill Store: A novel organization of reusable knowledge by applicability scope (Global, Domain, Competition), coupled with an LLM-driven promotion mechanism to abstract learnings upward.
2. Scoped Loading Mechanism: The first evaluation of "scoped loading" against "flat loading" (dumping all skills) and "empty loading" under a fixed skill inventory.
3. Empirical Validation on MLE-Bench Lite: Demonstration that a non-frontier model (Claude Sonnet 4.6) with a 12-hour budget can achieve competitive performance (77.3% medal rate) by leveraging organized knowledge, matching agents that use stronger models or longer budgets.

Experimental Results

Main Benchmark (MLE-Bench Lite)

• Performance: HASTE achieved a 77.3% medal rate (17/22 competitions) using Claude Sonnet 4.6 and a 12-hour budget. This matches the performance band of leading agents using frontier models (e.g., Gemini-3-Pro) and 24-hour budgets.
• Transfer Efficiency:
  • Cold Start: A single-pass run with no accumulated skills achieved only 40.9% medal rate.
  • Warm Start: Reloading global and domain skills lifted the same system to 77.3%, flipping 8 of 13 previously failed competitions to medal status.
  • Iteration Reduction: Warm starts required 52% fewer refinement iterations (7.8 vs. 16.3) to reach peak scores.
  • Hit Rate: The fraction of proposed changes kept by the agent rose from 42% (low inventory) to 85% (50+ skills).

Controlled Ablation (Fixed Inventory)

To isolate the effect of organization from skill volume, the authors ran 8 competitions with a fixed inventory of 159 skills under three conditions:

• Tiered Loading: Loaded only relevant skills (10–60 per task). Result: 100% medal rate.
• Flat Loading: Loaded all 159 skills (~145k characters). Result: 62.5% medal rate (same as empty loading).
• Empty Loading: No skills. Result: 62.5% medal rate.

Efficiency Findings:

• Flat loading consumed 2× more output tokens (3.78M) than empty loading (1.86M) for the same medal rate, and 2.7× more tokens per medal than tiered loading.
• Flat loading suffered from signal dilution (relevant skills buried in noise) and context budget displacement (crowding out reasoning).
• Tiered loading was most effective on harder or niche tasks (e.g., Audio, specific NLP tasks) where domain and competition-specific priors provided a known-good starting point.

Significance and Claims

The paper argues that better knowledge organization can partly substitute for model strength and compute budget in ML engineering agents. The primary claim is not merely that accumulating skills helps, but that the structure of that accumulation is critical.

• Organization over Volume: The ablation demonstrates that a large volume of skills is ineffective if loaded indiscriminately (flat loading). The hierarchical, scoped loading mechanism is the source of the performance gain.
• Efficiency: By narrowing the search space through structured priors, linear refinement loops become sufficient, reducing the need for complex tree-search or evolutionary strategies.
• Scalability: The system uses plain-text storage and simple concatenation, avoiding the theoretical limits of single-vector embeddings for multi-field conditional retrieval.

The authors conclude that HASTE represents a shift from "search-heavy" strategies to "knowledge-organization-heavy" strategies, enabling agents to solve complex engineering tasks more efficiently by avoiding the rediscovery of known facts.