TML-Bench: Benchmark for Data Science Agents on Tabular ML Tasks

This paper introduces TML-Bench, a benchmark for evaluating the end-to-end correctness and reliability of autonomous coding agents on Kaggle-style tabular machine learning tasks, demonstrating that the MiniMax-M2.1 model achieves the best aggregate performance across four competitions under varying time budgets.

The Gist

Imagine you are hiring a team of robot chefs to cook a meal using a specific set of ingredients (a spreadsheet of data). Your goal isn't just to see who can make the best dish if they get lucky once; you want to know who can consistently cook a delicious meal, every single time, within a strict time limit, without peeking at the recipe book or asking a friend for help.

This paper, TML-bench, is a giant cooking competition designed to test exactly that. Here is the breakdown in simple terms:

1. The Setting: The "Cook-Off"

In the world of data science, there are many "Kaggle competitions" where people try to predict things (like "Will this customer quit?" or "How many shoes will we sell?"). Usually, people just look at the final score.

But the author, Mykola Pinchuk, realized that's like judging a chef only on one lucky plate. What if the chef got lucky? What if they crashed the computer halfway through? What if they cheated by looking up the answer online?

TML-bench is a new, stricter test. It sets up a "clean kitchen" where:

• The Robots (AI Agents): 10 different AI models are the chefs.
• The Ingredients: Four real-world data problems (like predicting customer churn or foot traffic).
• The Rules: The robots have to load the data, clean it, train a model, and submit an answer.
• The Catch: They have no internet access (no Googling the answer) and their "knowledge" stops before the competition started (so they can't have memorized the test).

2. The Three Time Limits

The competition tests the robots under three different time pressures, like a cooking show with different rounds:

• The Sprint (240 seconds / 4 mins): Can they make a decent meal quickly?
• The Standard (600 seconds / 10 mins): Can they improve the dish with a bit more time?
• The Marathon (1200 seconds / 20 mins): Can they really refine the recipe if given plenty of time?

Note: The longer time limit also gave the robots a slightly different "instruction manual" focused on a specific cooking technique (XGBoost), making the test fairer.

3. The "Five-Taste" Rule (Reliability)

This is the most important part. In most tests, you run the AI once and see the score. In TML-bench, each robot cooks the same dish five times.

• Why? Because sometimes a robot might get a lucky break or a random glitch.
• The Score: The paper doesn't look at the "best" score. It looks at the median (the middle score of the five attempts).
• The Goal: If a robot gets a 10/10 once but fails four times, it's a bad chef. If it gets a 7/10 five times in a row, it's a reliable chef. The paper cares about consistency.

4. The Secret Sauce: The "Hidden Label"

How do they know if the robot actually learned the pattern or just guessed?

• The robots submit their answers to a "blind judge."
• The judge has a secret list of correct answers (the "private holdout") that the robots never saw.
• The robots only get a score based on how well they predicted these hidden answers. This prevents them from "cheating" by over-fitting to the data they were given.

5. The Results: Who Won?

After all the cooking, the paper found:

• The Star Chef: A model called MiniMax-M2.1-TEE was the most consistent and highest-scoring chef across all four competitions.
• Time Matters: Generally, giving the robots more time helped them cook better, but not always. Some robots got stuck in a loop or got confused, while others just got better.
• Reliability is Key: Some models had high scores but were very "jittery" (sometimes great, sometimes terrible). Others were steady. The paper argues that for real-world use, steady is better than lucky.

6. Why This Matters

Think of it like buying a car.

• Old Benchmarks: "This car hit 100 mph once on a perfect track!" (Great, but what if it breaks down every other day?)
• TML-bench: "This car drove 50 miles every day for a week, in the rain and sun, and never stalled."

The paper concludes that for AI to be useful in real businesses (like banks or retail), we need to stop celebrating "lucky runs" and start measuring reliability, consistency, and the ability to work under time pressure.

In a nutshell: TML-bench is a stress test for AI data scientists to prove they aren't just lucky guessers, but reliable workers who can get the job done, every time, without cheating.

Technical Summary

1. Problem Statement

Autonomous coding agents have demonstrated the ability to generate strong baselines for tabular machine learning tasks (e.g., Kaggle competitions) quickly. However, current evaluations often lack rigor regarding end-to-end correctness, reliability, and performance under strict time constraints. Existing benchmarks either focus on isolated coding tasks, lack private-holdout scoring (leading to potential data leakage), or do not account for run-to-run variability. Practitioners need a benchmark that evaluates not just the "best lucky run" but the reliability and consistency of agents across multiple attempts, different time budgets, and diverse tabular datasets.

2. Methodology

Benchmark Suite and Scope

• Tasks: The benchmark evaluates agents on four Kaggle-style tabular competitions (including bank-customer-churn, foot-traffic-wuerzburg, and two playground-series tasks).
• Time Budgets: Three distinct wall-clock time limits are tested per task: 240s, 600s, and 1200s.
• Instruction Sets: The 1200s budget utilizes a specific XGBoost-focused instruction set, while shorter budgets use fixed instructions appropriate for their time constraints.
• Models: The study evaluates 10 Open-Source Large Language Models (LLMs).
• Coverage Requirement: To be included in the main results, a model must achieve complete coverage: 5 successful runs for every combination of task and time budget (12 settings total).

Evaluation Protocol

• Execution Environment: Runs are executed in a clean, per-run workspace managed by the Kilo Code harness.
• Scoring Mechanism:
  • Private-Holdout Scoring: Agents submit predictions to a hidden test set (holdout labels) that the agent cannot access. The final score is calculated on this private data.
  • Success Criteria: A run is successful only if it produces a valid submission file and a score on the hidden labels.
• Contamination Controls:
  • Internet Access: Disabled during all benchmark runs.
  • Knowledge Cutoff: All evaluated models have pretraining cutoff dates predating the start of the tested competitions (earliest competition: late October 2025).
• Aggregation Strategy:
  • For each (Model, Task, Budget) setting, the reported metric is the median of the 5 successful runs.
  • Normalization: To compare across different metrics (e.g., AUC vs. RMSE), a within-setting min-max normalization is applied. Scores are converted to a "higher is better" direction, then scaled so the best model in that setting gets 1.0 and the worst gets 0.0.
  • Primary Leaderboard: Aggregates the "best budget per competition" normalized scores, averaged equally across the four competitions.

3. Key Contributions

1. Strict Benchmark Protocol: Introduces a deterministic, repeatable protocol for Kaggle-style tabular tasks featuring private-holdout scoring and strict submission validation.
2. Reliability-Focused Reporting: Shifts focus from single best attempts to median performance, success rates, and run-to-run variability (stability).
3. Contamination-Aware Design: Implements strict controls (offline execution, knowledge-cutoff verification) to ensure results reflect genuine agent capability rather than memorized competition data.
4. Low-Cost, Accessible Setup: The evaluation is designed to be runnable by individual practitioners with a marginal cost of approximately $10 for the full suite.
5. Reproducibility: Provides full code, run logs, and scripts to regenerate all figures and tables.

4. Key Results

Aggregate Performance

• Top Performer: MiniMax-M2.1-TEE achieved the best aggregate performance score across all four competitions under the primary aggregation method.
• Time Scaling: Generally, performance improves as the time budget increases from 240s to 1200s. However, scaling is "noisy" for some individual models, indicating that more runs may be needed for stable scaling curves.

Reliability and Stability

• Variability: There is significant variation in reliability even among top-performing models. Some models show high success rates and low variance (tight Interquartile Ranges), while others exhibit high failure rates or unstable scores.
• Failure Modes: Specific models showed large outliers in specific settings (e.g., GLM 4.7 Flash on the foot-traffic task at 1200s had a median RMSE of 0.107 with a massive IQR, indicating instability).

Cross-Competition Consistency

• Model rankings vary across different competitions. The paper uses rank standard deviation to quantify consistency; models with lower deviation are more robust across different data distributions.

Specific Task Highlights

• Bank Customer Churn (AUC): Top models clustered tightly around 0.926–0.928 at 1200s, though some models (e.g., NVIDIA-Nemotron-3-Nano) performed significantly worse.
• Playground Series (RMSE): At 1200s, medians were tightly clustered, but specific models showed large failure-mode outliers at lower budgets (e.g., TNG-R1T2-Chimera at 600s).

5. Significance and Implications

• Beyond "Best Run": TML-bench demonstrates that capability on a single lucky run is insufficient for practical deployment. Reliability and stability are critical metrics for data science agents.
• Trade-off Visibility: The benchmark makes the trade-offs between speed (240s) and iterative improvement (1200s) visible, helping practitioners choose agents based on their specific latency requirements.
• Standardization: By enforcing private-holdout scoring and contamination controls, TML-bench sets a new standard for evaluating autonomous data science agents, reducing the risk of inflated scores due to data leakage.
• Practical Utility: The low cost and open-source nature of the benchmark make it accessible for individual researchers and practitioners to test and iterate on agent workflows without massive infrastructure costs.

6. Limitations

• Token Accounting: The current logs do not track token consumption, only max token configuration. Token efficiency analysis is deferred to future revisions.
• Instruction Coupling: The 1200s budget uses a different instruction set (XGBoost-focused) than shorter budgets, meaning performance gains are a mix of increased time and different prompting strategies.
• Run Count: While 5 runs per setting are sufficient for aggregate trends, they may be noisy for analyzing scaling curves at the individual model level.