💻 computer science

Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

This paper introduces Repeated Optimizer Resampling (ROR), a method that dynamically selects the best optimizer during a single training run by periodically scouting candidate optimizers for short intervals, thereby achieving performance comparable to exhaustive fixed-optimizer searches while using significantly less computational resources.

Original authors: Ronald Richman, Mario V. Wüthrich

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ronald Richman, Mario V. Wüthrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Deep learning, the technology behind everything from voice assistants to medical imaging, relies on artificial brains called neural networks. To teach these networks, researchers must choose a mathematical tool known as an optimizer. Think of an optimizer as the set of rules a student uses to correct their mistakes while studying; it decides how much to adjust their understanding after every new piece of information. For years, the standard practice has been to pick one set of rules at the very beginning of training and stick with it until the job is done. This choice is often made by guesswork or habit, yet it is a critical decision that can determine whether the final model is brilliant or merely average. The problem is that the best set of rules for the start of a lesson might not be the best for the end, and trying to find the perfect rule by testing every possibility one by one is incredibly expensive, requiring vast amounts of time and computing power that often go to waste.

A team of researchers from insureAI and ETH Zürich set out to see if they could make this process smarter and cheaper. Instead of locking in a single optimizer before training begins, they developed a method called Repeated Optimizer Resampling, or ROR. Imagine a long race where, instead of assigning one runner to the whole course, a coach checks in every few miles. At each checkpoint, the coach sends out a small team of different runners, each using a different strategy, to run just a short distance ahead. The coach watches who performs best over that short stretch, keeps that runner, and sends them on for the next leg of the race, while the others are sent home. This process repeats throughout the entire training session, allowing the team to switch strategies as the journey progresses. The researchers tested this idea on four different tasks: two involving classifying images of handwritten numbers and clothing, and two involving predicting insurance claims from complex data tables.

The results showed that this dynamic approach works remarkably well, but with a surprising twist regarding how much effort is needed. The researchers found that the short "scouting" runs did not need to be long to be effective. In fact, sending the runners out for just one step before deciding who to keep was enough to find a path that performed almost as well as the best possible fixed strategy found by testing every option exhaustively. By using this one-step scouting method, the team used only about a quarter to a third of the total computing power required to run all nine different strategies to completion. This means they achieved nearly the same high-quality results while saving a massive amount of time and energy. The method was able to identify that different tasks required different strategies; for example, one image task favored a specific optimizer from start to finish, while an insurance model switched strategies several times as it learned, proving that a single fixed rule is not always the best choice.

The study also compared two ways of handling the "memory" of the optimizer. In one version, if the same strategy won two rounds in a row, it kept its accumulated knowledge and momentum. In the other, every time a strategy was picked, it started with a fresh, clean slate. The researchers found that keeping the memory did not consistently lead to better results or lower costs. The most important discovery was that the length of the scouting period mattered far more for the cost than for the final performance. Because the most significant learning happens in the very first steps of training, a brief check is sufficient to spot the most promising direction. While a single early check, known as a one-shot selection, was cheaper and worked well for the image tasks where the best strategy stayed the same, the repeated checks of ROR proved valuable for the insurance models where the best strategy changed over time.

Ultimately, the research suggests that we do not need to exhaustively test every possibility to find a good training path, nor do we need to stick rigidly to a single choice. By allowing the training process to adapt and switch strategies based on short, frequent checks, we can reach high performance with a fraction of the usual cost. The method does not guarantee a result better than the absolute best fixed strategy found after a full search, but it gets very close to that peak performance while using significantly fewer resources. This offers a practical way to navigate the complex landscape of training artificial intelligence, showing that a flexible, adaptive approach can be just as effective as a massive, exhaustive search, provided the checks are frequent and the decisions are made quickly.

Technical Summary: Repeated Resampling for Adaptive Optimizer Selection

Problem Statement
Training deep neural networks typically requires selecting a single optimizer before training begins, which is then kept fixed for the entire duration. While optimizer choice is a critical hyperparameter, the optimal selection for a specific problem is rarely known in advance. Identifying the best fixed optimizer retrospectively requires an exhaustive search—training every candidate to convergence and discarding all but the winner—which is computationally expensive. Furthermore, the optimizer best suited for early descent may differ from the one optimal for late-stage refinement, suggesting that a static choice may be suboptimal.

Methodology: Repeated Optimizer Resampling (ROR)
The paper proposes Repeated Optimizer Resampling (ROR), a procedure that searches for the optimal optimizer during a single evolving training run rather than before it. ROR operates on two time scales: a scouting period ( $s$ ) and a retained segment length ( $b$ ), where $1 \le s \le b$ .

Tournament Mechanism: At regular intervals, a tournament is initiated. The current model weights are copied $K$ times (once for each candidate optimizer).
Scouting: Each candidate optimizer trains its copy for $s$ epochs using the same data order.
Selection: The optimizer yielding the best validation objective ( $J_{val}$ ) is declared the "scout winner."
Continuation: The winner continues training from its scout weights for the remaining $b-s$ epochs.
Commitment: If the completed segment improves the incumbent validation objective by a threshold $\delta$ , the new weights and optimizer state become the incumbent. Otherwise, training stops.

State Policies
The paper evaluates two variants of ROR regarding how optimizer state (e.g., momentum buffers, adaptive moment estimates) is handled:

State-Preserving ROR (SP-ROR): If the incumbent optimizer wins again, its accumulated state is preserved and resumed. Challengers start with newly initialized state.
Cold-Start ROR (CS-ROR): Every optimizer, including the incumbent, is reinitialized with new state at the start of every tournament.

This approach differs from methods like Hyperband, which permanently eliminate losing configurations. In ROR, every optimizer re-enters each tournament from the current incumbent weights, allowing a candidate that loses early to be selected later.

Experimental Design
The authors evaluated nine fixed optimizers (including Adam, AdamW, Lion, Muon, SGD-Nesterov, Nadam, RMSprop, LAMB, and ScheduleFreeAdamW) across four tasks:

Image Classification: MNIST and Fashion-MNIST (fully connected MLPs).
Insurance Pricing: Motor third-party liability claim counts using Poisson deviance, evaluated on both an MLP and a Transformer architecture.

Experiments used ten random seeds. The "oracle" benchmark was defined by the best fixed optimizer found after exhaustive comparison. Costs were measured in epoch-equivalents, accounting for all training branches (accepted and rejected).

Key Results

Cost Efficiency: One-epoch ROR ( $s=1$ ) requires only 24% to 35% of the aggregate training cost needed to identify the best fixed optimizer exhaustively. It is also 51% to 68% cheaper than ROR with longer scouting periods ( $s=3$ ).
Predictive Performance: Across all four tasks, no ROR variant showed a statistically significant difference (at 95% confidence) from the best fixed optimizer found in hindsight.
- On MNIST, one-epoch ROR achieved slightly lower accuracy than the best fixed optimizer (Muon) but at a fraction of the cost.
- On Fashion-MNIST and the Insurance Transformer, one-epoch ROR achieved performance comparable to the best fixed optimizers, with no paired comparisons excluding zero.
- On the Insurance MLP, specific ROR configurations achieved lower deviance than the best fixed optimizer (Lion), but paired comparisons with $s=3$ (and $s=1$ ) all included zero, indicating no clear evidence of improvement.
Scouting Duration: Increasing the scouting period ( $s$ ) from 1 to 3 epochs significantly increased computational cost without yielding consistent predictive improvements. The authors attribute this to the fact that most validation loss reduction occurs in the earliest optimization steps, after which losses tend to flatten.
Optimizer Dynamics: The resulting optimizer schedules were not static. On the Insurance MLP, the preferred optimizer changed frequently (averaging 5.6 switches per run), whereas image tasks were more stable. This confirms that the optimal optimizer can change as training progresses.
State Preservation: Comparing SP-ROR and CS-ROR revealed no consistent predictive or computational advantage for preserving state. The choice of state policy did not significantly alter the final model performance or the frequency of optimizer switches.

Significance and Claims
The paper claims that ROR provides a practical mechanism to search over optimizers without completing every candidate run to convergence.

Adaptive Selection: It demonstrates that the preferred optimizer can change during training, supporting the hypothesis that adaptive sequences could outperform fixed choices, or at least match the best fixed result without the exhaustive search cost.
Practicality: The primary contribution is the demonstration that short scouting ( $s=1$ ) is sufficient to capture most of the value of optimizer selection. This allows practitioners to approach the performance of the best fixed optimizer (found in hindsight) using a fraction of the training budget.
Baseline Comparison: The "one-shot" baseline (selecting an optimizer once near the start) was cheaper than one-epoch ROR and performed well on MNIST. However, on tasks where the optimal optimizer shifts later in training (e.g., Insurance MLP), repeated tournaments provided better results. The paper concludes that the value of repeated tournaments depends on the task's sensitivity to optimizer changes over time.

The authors remain modest, noting that ROR did not consistently outperform the best fixed optimizer in a statistically significant manner across all tasks, but it successfully approximated the best fixed result at a significantly lower cost. The study highlights that while longer scouting provides more evidence per tournament, it is often unnecessary given the diminishing returns of early training steps.

Technical Summary: Repeated Resampling for Adaptive Optimizer Selection

More like this