💻 computer science

KompeteAI: Accelerated Autonomous Multi-Agent System for End-to-End Pipeline Generation for Machine Learning Problems

KompeteAI is a novel AutoML framework that accelerates end-to-end pipeline generation by integrating RAG for diverse idea sourcing, a merging stage for recombining strong solutions, and predictive scoring to bypass costly execution cycles, thereby outperforming existing state-of-the-art methods on MLE-Bench and the proposed Kompete-bench.

Original authors: Stepan Kulibaba, Artem Dzhalilov, Roman Pakhomov, Oleg Svidchenko, Alexander Gasnikov, Aleksei Shpilman

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Stepan Kulibaba, Artem Dzhalilov, Roman Pakhomov, Oleg Svidchenko, Alexander Gasnikov, Aleksei Shpilman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the perfect recipe for a complex dish, but you have to do it alone, and every time you try a new ingredient, you have to cook the entire meal from scratch to see if it tastes good. This takes hours, and if you make a mistake in the first step, you have to throw away the whole pot and start over.

This is exactly the problem with current AutoML (Automatic Machine Learning) systems. They try to build AI models automatically, but they are slow, repetitive, and often get stuck because they can't easily mix and match good ideas.

Enter KompeteAI. Think of it as a super-charged, collaborative cooking team that doesn't just cook one meal at a time, but runs a massive, organized kitchen where chefs specialize, share secrets, and taste-test before cooking the full meal.

Here is how KompeteAI works, broken down into simple concepts:

1. The Problem: The "One-Shot" Trap and the "Slow Cook"

Current AI systems usually try to generate a whole solution in one go (like a chef guessing the whole recipe at once). If they get one spice wrong, the whole dish is ruined.
Other systems try to build a tree of ideas (like trying every possible combination of spices), but they are slow because they have to actually cook the full meal to test if a spice works. This creates a bottleneck: they spend 90% of their time cooking and only 10% thinking.

2. The Solution: A Team of Specialized Chefs (Multi-Agent System)

KompeteAI doesn't use one brain; it uses a team of specialized AI agents, each with a specific job:

The Researcher (Insighter): This agent looks at the ingredients (data) and suggests new ideas. It doesn't just guess; it has a magic library (RAG) where it can instantly pull up winning recipes from past competitions and scientific papers. It's like having a chef who can instantly read the world's best cookbooks while you are cooking.
The Builder (Coder): Turns the idea into actual code.
The Inspector (Checker): Makes sure the code makes sense and doesn't have logical holes.
The Fixer (Debugger): If the stove catches fire (code error), this agent fixes it quickly without burning the whole kitchen down.

3. The Secret Sauce: "Merging" and "Adding"

Instead of just trying random new recipes, KompeteAI has two superpowers:

Adding: It constantly brings in fresh ideas from its "magic library" (Kaggle and arXiv) to keep the menu diverse.
Merging: This is the game-changer. Imagine you have two great partial dishes: one has the perfect sauce, and the other has the perfect seasoning. Instead of picking one, KompeteAI merges them into a new, super-dish. It takes the best parts of different solutions and combines them to create something better than either could be alone.

4. The Speed Hack: The "Taste-Test" Model

The biggest problem in cooking is that you have to wait hours for the food to cook to know if it's good. KompeteAI solves this with a Predictive Scoring Model.

The Analogy: Imagine a "taste-tester" who can look at the raw ingredients and the chef's notes and say, "I'm 90% sure this will be a 5-star dish," without you actually having to cook it.
This allows the system to discard bad ideas instantly and focus only on the promising ones. This speeds up the process by 6.9 times, meaning they can try way more recipes in the same amount of time.

5. The New Arena: Kompete-bench

The paper also points out that the current "sports league" for these AI systems (called MLE-Bench) is flawed. It's like judging a race by looking at a map of the track rather than the actual finish line, or using a track that is too big to run on.
The authors created Kompete-bench, a new, fairer arena. It uses real, recent competitions where the AI has to compete against actual humans, ensuring the test is about genuine problem-solving, not just memorizing old answers.

The Result

In this new arena, KompeteAI didn't just participate; it won. It beat the previous best AI systems by a significant margin (about 3% better on average) and managed to outperform human participants in 14% of the cases.

In summary: KompeteAI is a smart, fast, collaborative team of AI agents that reads the best books, mixes the best ideas together, and uses a "crystal ball" to predict which ideas will work before wasting time cooking them. It turns the slow, lonely process of building AI into a fast, collaborative, and highly successful team sport.

1. Problem Statement

The paper addresses critical limitations in current Large Language Model (LLM)-based AutoML systems, specifically regarding exploration strategies and execution bottlenecks:

Constrained Exploration: Existing methods often rely on "one-shot" generation (lacking iterative refinement) or Monte Carlo Tree Search (MCTS) that struggles to recombine promising partial solutions from different branches. This leads to the premature discarding of valuable solution components.
Static Knowledge: While Retrieval-Augmented Generation (RAG) is used, most systems only apply it in early stages, leading to "knowledge decay" where the system fails to incorporate fresh, domain-specific insights (e.g., from recent Kaggle notebooks or arXiv papers) during later optimization phases.
Execution Bottleneck: Validating a single pipeline requires full code execution and training, which can take hours. Debugging late-stage errors forces complete retraining, creating a severe scalability issue that stifles iterative refinement.
Benchmark Bias: The primary benchmark, MLE-Bench, suffers from evaluation bias because it constructs test sets by partitioning original training data and compares results against private leaderboards, which does not reflect true generalization or real-world performance.

2. Methodology: KompeteAI

KompeteAI is an autonomous multi-agent framework designed for structured, multi-stage pipeline generation. It operates through three main stages: Pipeline Setup, Tree Initialization, and Tree-Guided Exploration.

A. Architecture & Agents

The system employs a multi-agent architecture where the ML workflow is decomposed into discrete stages: Exploratory Data Analysis (EDA), Feature Engineering (FE), and Model Training (MT).

Insighter Agent: Generates candidate ideas. It utilizes Tree Memory (analyzing past nodes via embedding similarity to ensure diversity) and Adaptive RAG. Unlike static RAG, the Insighter queries external sources (Kaggle, arXiv) on-demand when it encounters creative impasses.
Checker Agent: Validates logical consistency and schema correctness using unit tests.
Coder Agent: Translates validated ideas into executable code, optimizing hyperparameters and architecture for resource constraints.
Debugger Agent: Uses an accelerated debugging loop (simplified code, smaller data samples, reduced iterations) to detect errors quickly without full execution.
Metric & Validator Agents: Handle data partitioning, metric definition, and submission validation.

B. Core Operators: Adding and Merging

The framework explores the solution space using two novel operators:

Adding: Dynamically expands the search tree by generating new FE and MT nodes based on the current context vector (combining EDA insights, metadata, and external knowledge).
Merging: A controlled mechanism to combine the most successful solutions.
- It merges FE nodes (recombining structural/statistical traits) and MT nodes.
- It employs dual memory buffers (short-term and long-term) to track failed merges, preventing redundant or destructive combinations and enabling adaptive learning.
- Unlike prior methods that merge entire pipelines at the end, KompeteAI performs bottom-up, multi-level merging throughout the search, allowing for synergistic combinations of feature engineering and model architectures.

C. Accelerated Evaluation (Scoring Model)

To overcome the execution bottleneck, KompeteAI introduces a Predictive Scoring Model:

Mechanism: Instead of full training, the model predicts the final score of a candidate pipeline using an LLM.
Inputs: It takes the candidate's description, early-stage training logs (reduced epochs), metrics from a shared reference architecture trained on the same features, and "anchor ideas" (precomputed examples of similar models).
Benefit: This allows the system to prune weak solutions early and prioritize promising candidates, accelerating the evaluation process by 6.9x.

3. Key Contributions

Stage-Decomposed Multi-Agent Architecture: A framework that partitions the ML workflow, enabling agents to specialize. It introduces dynamic RAG for continuous knowledge injection and controlled merging operators to systematically recombine optimal partial solutions.
Accelerated Evaluation Paradigm: A two-part solution combining a predictive scoring model (LLM-based performance estimation) and an accelerated debugging framework (simplified execution) to drastically reduce validation time.
Kompete-bench: A new benchmark designed to address the limitations of MLE-Bench. It consists of 26 competitions (15 from MLE-Bench Lite and 11 recent 2024–2025 competitions) with strict data size limits and evaluation based on real Kaggle leaderboards to eliminate evaluation bias.

4. Experimental Results

The authors evaluated KompeteAI against state-of-the-art baselines (AIDE, RD-Agent, ML-Master, MLE-STAR).

MLE-Bench Performance: KompeteAI achieved a 51.5% medal rate on the MLE-Bench Lite subset, outperforming the next best method (ML-Master at 48.5%) by an average of 3%. Statistical significance was confirmed via paired t-tests ( $p \le 0.05$ ).
Real-World Validation: When tested on actual Kaggle leaderboards, a significant gap was observed between MLE-Bench scores and real performance (e.g., RD-Agent dropped from 27% on MLE-Bench to 13% on real leaderboards), highlighting the bias in the original benchmark.
Kompete-bench Performance: On the "Contemporary" subset of Kompete-bench (where no agent won medals), KompeteAI demonstrated the ability to beat 14% of human participants, significantly outperforming all baselines.
Efficiency: The acceleration framework enabled the system to execute 6.9 times more iterations within the same time budget compared to a non-accelerated baseline.
Ablation Studies:
- Removing RAG caused a sharp performance drop on contemporary tasks (14% $\to$ 11.6%), proving the necessity of external knowledge for new problems.
- Removing Merging caused a massive drop (14% $\to$ 9.5%), confirming that recombining partial solutions is critical for high performance.
- Removing the Scoring Model reduced performance (14% $\to$ 7.4%), validating the importance of efficient exploration.

5. Significance

Advancement in AutoML: KompeteAI sets a new state-of-the-art by effectively combining LLM reasoning with algorithmic search (MCTS-like structures) and dynamic knowledge retrieval.
Solving the Scalability Problem: The proposed predictive scoring and accelerated debugging paradigms offer a generalizable solution to the execution bottleneck, making complex, iterative AutoML feasible within practical timeframes.
Benchmark Integrity: By introducing Kompete-bench, the paper highlights the flaws in current evaluation metrics (MLE-Bench) and provides a more rigorous, unbiased standard for evaluating autonomous ML agents against real-world human performance.
Future Directions: The work suggests a path toward agents that can contribute to scientific discovery by handling hypothesis generation and experimental design, moving beyond just competition-style tasks.