📊 statistics

Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons

This paper demonstrates that short-budget AutoML comparisons are frequently inflated by protocol defects such as test-set selection bias and unenforced time limits, showing how correcting these issues in a specific case study caused a previously dominant system's win rate to collapse and eliminated statistically significant advantages over competitors.

Original authors: Guilin Zhang, Kai Zhao

Published 2026-08-10

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Guilin Zhang, Kai Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where scientists build "learning machines" that can look at a spreadsheet of data—like a list of house prices or medical records—and figure out how to predict the future. This field is called AutoML (Automated Machine Learning). The goal is to let a computer try thousands of different math tricks on its own to find the one that works best. Usually, these machines are tested in long, careful marathons that take hours, where they are strictly watched to make sure they don't violate the rules. But in the real world, developers often want quick results. They run these machines for just 30 or 60 seconds to see which one is the fastest winner. It's like a sprint instead of a marathon. The problem is, when you run a race that fast, it's very easy to accidentally break the rules without even realizing it, making a slow, clumsy runner look like a world-record sprinter.

This paper tells the story of a researcher who built their own tiny, simple learning machine called "Orcetra." They pitted it against two famous, heavy-duty champions (FLAML and AutoGluon) in a series of 513 quick 60-second races. At first glance, the results looked amazing: Orcetra won more than half the races, beating the giants by huge margins. The numbers were so perfect, and the math so convincing, that it looked like a massive breakthrough. But the authors realized they had been tricked by their own experiment. They discovered that Orcetra wasn't actually smarter; it was just violating the rules in two very sneaky ways. First, it was peeking at the "answer key" (the test data) while it was still practicing, allowing it to pick the winner based on luck rather than skill. Second, it was ignoring the stopwatch, running for twice as long as the other machines while pretending to stop at the same time. When the researchers fixed the rules, stopped the peeking, and enforced the time limit, Orcetra's "superpowers" vanished. It went from being a champion to being just average, proving that in the world of quick AI tests, how you measure the race matters just as much as how fast you run.

The Setup: A Race with a Broken Stopwatch

To understand the story, let's look at the three racers. FLAML and AutoGluon are like professional athletes who train for hours. They are complex, powerful systems designed to find the best solution over a long period. Orcetra, the system built by the paper's authors, is the underdog. It's a tiny, simple script written in just 1,661 lines of code. It doesn't have fancy tricks; it just tries a bunch of standard math models, picks the best one, and moves on.

The researchers set up a "sprint" competition. They took 513 different datasets (collections of data) and gave every machine exactly 60 seconds to find the best prediction. In the first round, using their original rules, Orcetra looked like a miracle. It won 57.1% of the races against the other two, while the giants only won 21.6% and 10.9% respectively. Against just FLAML in a 30-second sprint, Orcetra won a staggering 78.4% of the time. The math said this was a huge, undeniable victory.

But the authors suspected something was wrong. They decided to audit their own experiment, looking for the "ghosts" in the machine that might have caused these fake wins.

The First Flaw: The "Peeking" Problem

The first major flaw was how Orcetra chose its winner. Imagine a student taking a practice test. A fair student studies, takes the test, and then gets a score. But Orcetra was doing something different. It was taking the practice test, looking at the answers, and then taking the test again, and again, and again.

In technical terms, the machine was scoring every single model it tried against the test split (the data meant to be the final exam). It tried dozens of models in those 60 seconds, checked how well they did on the test data, and then simply reported the best score it ever saw. This is a classic trap called selection bias.

Think of it like rolling a die. If you roll a die once, you get a random number. But if you roll it 50 times and only show me the highest number you got, I might think you have a magic die that always rolls sixes. In reality, you just rolled enough times to get lucky. Because Orcetra tried so many models and picked the "luckiest" one based on the test data, its score was artificially inflated. The other machines, FLAML and AutoGluon, were honest: they picked their best model using only the practice data (training data) and only looked at the test data once at the very end.

The Second Flaw: The "Loophole" Stopwatch

The second flaw was even more physical. The experiment had a rule: "Stop after 60 seconds."

FLAML and AutoGluon respected this rule. If they were in the middle of a calculation when the timer hit 60, they would stop or try to finish quickly. They were disciplined. Orcetra, however, had a bug in its logic. It checked the time before it started a new task, but once it started a task, it didn't care if the timer ran out. It would let a calculation run for as long as it wanted.

The data showed that while the budget was set to 60 seconds, Orcetra actually ran for a median of 120 seconds—twice as long as the limit! In fact, it exceeded the time limit on 78% of the datasets. Meanwhile, the other machines stuck close to the 60-second mark. It's like a race where the other runners stop at the finish line, but Orcetra keeps running for another lap, giving it way more time to solve the problem. The "60-second" label was just a suggestion to Orcetra, not a rule.

The Third Glitch: The "Spliced" Results

There was a third, smaller issue found while writing the paper. The researchers had run a second experiment later on, but they accidentally mixed the results of the first run with the second run when they were counting the wins. This "splicing" made Orcetra look even better than it was, pushing its win rate up to 61.2% in some counts. When they separated the runs, the numbers dropped back down. This was a simple human error, but it showed how easily results can be skewed if you aren't careful with your files.

The Great Correction: Running the Race Fairly

To find the truth, the authors re-ran the entire experiment with a new, strict protocol. They fixed the three problems:

No more peeking: Orcetra had to pick its best model using a "validation" set (a practice exam), and then only look at the "test" set (the final exam) one single time at the very end, just like the other machines.
Strict time limits: They used an external timer to kill the process if it went over 60 seconds, ensuring no one got extra time.
Fair resources: They made sure all machines got the exact same amount of computer power so no one was faster just because they had more cores.

The Result: The Magic Disappears

When they ran the race fairly, the "miracle" vanished.

Original (Violating Rules) Win Rate: Orcetra won 59.4% of the re-run subset.
Corrected (Fair) Win Rate: Orcetra's win rate plummeted to 34.3%.

In the fair race, Orcetra was no longer beating the giants. It won 34.3% of the time, while FLAML won 28.0% and AutoGluon won 27.3%. The difference was so small that it could easily be just random chance. The statistical "significance" that had made the original results look like a massive discovery disappeared completely. The p-values (a measure of how likely a result is to happen by luck) went from being astronomically small (like 9.5 × 10⁻⁴⁶) to being totally unremarkable (like 0.39), which is basically a coin flip.

What Did We Learn?

The paper concludes that the "win" was an illusion created by two main factors:

Peeking (Selection Bias): This accounted for about 4.8 percentage points of the win rate. It was real, but small.
Extra Time (Compute): This was the big one. Because Orcetra ran for twice as long as the others, it got to try more models. This unfair advantage accounted for the rest of the massive win rate.

The authors also measured exactly how much "peeking" helps. They found that while peeking does give a slight boost, it's not as huge as some math theories predicted. Because all the models were tested on the same data, the "luck" canceled out a lot. The boost was about 0.27 accuracy points, which is tiny compared to the massive gaps they originally saw.

The Takeaway for Everyone

This story isn't just about one piece of code; it's a warning for anyone comparing fast AI tools. If you run a 60-second test, you have to be incredibly careful.

Don't let the test data peek: Make sure the machine picks its winner before seeing the final answers.
Enforce the time limit: If you say 60 seconds, make sure it stops at 60 seconds.
Check the clock: Always report how long the machine actually ran, not just how long you told it to run.

The paper ends with a checklist for anyone doing these quick comparisons. It suggests that if you see a system winning by a huge margin in a short time, it's probably not because it's smarter—it's probably because it violated the rules. The "winner" of the original race wasn't a genius; it was just a runner who ignored the referee's whistle.

Technical Summary: "Winning by Peeking"

Problem Statement

Short-budget AutoML comparisons (tens of seconds per dataset) are ubiquitous in tool READMEs and workshop papers but are prone to severe protocol errors that are invisible in standard results tables. This paper presents a case study of the authors' own system, Orcetra, which appeared to significantly outperform established systems (FLAML and AutoGluon) on 513 OpenML datasets under a nominal 60-second budget. The reported win rates (57.1% overall, 78.4% against FLAML at 30 seconds) were driven not by superior search capabilities, but by three specific protocol defects:

Selection on the Test Set: The search loop evaluated every candidate model on the held-out test split and reported the maximum score, effectively overfitting to the noise in a single test split. Competitors used cross-validation on training data and touched the test set only once.
Unenforced Budgets: The system checked the time budget only before launching a candidate model. Once launched, a candidate was allowed to run to completion regardless of the deadline. Consequently, Orcetra consumed a median of 120 seconds per dataset against a 60-second budget (2.24× the time used by AutoGluon).
Result Splicing: A later regression-only re-run was inadvertently merged with the original results by deduplicating on dataset ID, artificially inflating the win rate further.

Methodology

The paper employs a rigorous audit and re-evaluation strategy to isolate and measure the impact of these defects.

The System (Orcetra)

Orcetra is a 1,661-line Python AutoML engine that performs guided random search over a fixed pool of scikit-learn models (e.g., Random Forest, Gradient Boosting, XGBoost). It includes a deduplication cache and a post-hoc weighted ensemble. The system was designed to be simple and reproducible, with no neural architecture search or meta-learning.

Experimental Setup

Datasets: 513 OpenML datasets (382 classification, 131 regression) with 50–100,000 instances.
Baselines: FLAML (2.5.0) and AutoGluon (1.5.0).
Original Protocol: 60-second nominal budget, 80/20 train/test split, selection based on test-set performance.
Corrected Protocol:
- Selection: The training split is further divided into a search-training set (64%) and a validation set (16%). Selection occurs on the validation set. The selected model is then refitted on the full training set (80%) and evaluated exactly once on the test set.
- Budget Enforcement: The search runs in a child process with an external timeout. If the budget is exceeded, the process is killed.
- Compute Control: All frameworks are pinned to equal shares of the machine to prevent contention artifacts.

Measurement Strategy

To attribute the performance collapse, the authors recorded three estimands during a single corrected run on a subset of datasets:

A (Select on Test): The maximum test score observed during the search (simulating the original defect).
B (Select on Validation, Same Data): The test score of the model chosen by validation, fitted on the reduced training set.
C (Select on Validation, Refit): The test score of the model chosen by validation, refitted on the full training set (the fair comparison).

Key Contributions

Quantification of Protocol Defects: The paper demonstrates that the original "win" was an artifact. Under the corrected protocol, Orcetra's win rate on the re-run subset dropped from 59.4% to 34.3%, rendering it statistically indistinguishable from FLAML and AutoGluon.
Attribution of Bias: By comparing estimands A, B, and C, the authors isolated the sources of the inflated margin:
- Selection Bias: Moving selection from the test set to the validation set accounted for 4.8 percentage points of the win rate drop.
- Compute Inequality: The remainder of the drop (approx. 20 points) was attributed to the unenforced budget, which allowed Orcetra to consume significantly more compute time than its competitors.
Empirical Selection Curve: The authors measured the selection bias as a function of the number of candidates ( $K$ ). While the bias grows with $K$ , the realized bias reached only 0.27 accuracy points at the largest budget. This is approximately five times smaller than the theoretical upper bound ( $\sigma\sqrt{2 \ln K}$ ) derived from marginal standard error, because candidates scored on shared test rows cancel out common noise.
Checklist for Short-Budget Comparisons: The paper provides a practical checklist for researchers, including:
- Reporting realized wall-clock time, not just the nominal budget.
- Logging the number of candidates evaluated to assess noise floors.
- Tracing evaluation data to ensure the test set is read exactly once for decision-making.
- Enforcing budgets externally (e.g., via process killing) rather than checking between iterations.
- Using paired, same-run measurements to isolate protocol changes from run-to-run variance.

Results

Original Findings (Invalid): Orcetra won 57.1% of datasets against 21.6% for AutoGluon and 10.9% for FLAML (60s budget). Against FLAML alone at 30s, it won 78.4%.
Corrected Findings: On a stratified subset of 143 datasets, Orcetra won 34.3%, FLAML won 28.0%, and AutoGluon won 27.3%. Pairwise comparisons showed no significant difference ( $p > 0.1$ ).
Wall-Clock Discrepancy: Under the original protocol, Orcetra's median runtime was 120s (vs. 70s for FLAML and 62s for AutoGluon). Under the corrected protocol, Orcetra's median runtime dropped to 72s, but it still exceeded the 60s budget on 100% of datasets due to the inability to interrupt a running fit.
Prediction Market Case Study: The paper includes a secondary case study where Orcetra was applied to Polymarket prediction markets. The results showed a "favourite-longshot" bias leaning in the expected direction but failing to reach statistical significance, illustrating the difficulty of distinguishing signal from noise in small-sample regimes.

Significance and Claims

The paper claims that the specific regime of short-budget AutoML comparisons is where protocol errors cause the most damage and are least likely to be detected. The significance lies not in proposing a new AutoML algorithm, but in exposing how easily a system can "win" a benchmark through protocol violations rather than algorithmic superiority.

The authors emphasize that the distortion is hidden by conventional results tables, which report aggregate win rates without revealing the underlying compute consumption or selection mechanics. The paper argues that the "noise floor" in short-budget, single-split evaluations is high, and the number of selection events (candidates) is large, creating a perfect storm for selection bias.

The work serves as a self-critique and a methodological guide, urging the community to adopt "honest" protocols where the test set is touched exactly once, budgets are enforced externally, and compute resources are equalized. The authors conclude that the specific margins reported in informal comparisons are often below the resolvable noise floor, making them unreliable indicators of true algorithmic progress.