📊 statistics

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

This paper demonstrates that combining weight averaging with simple, horizon-free learning rate schedules (such as polynomial decay or $1/\sqrt{t}$ ) achieves final performance comparable to tuned cosine schedules for large language models, offering a practical and theoretically grounded alternative for training in open-ended settings where the total horizon is unknown.

Original authors: Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, large language models are the engines that power everything from writing assistants to complex reasoning tools. To make these engines run, researchers must teach them by feeding them vast amounts of text, a process known as pretraining. This teaching requires a careful balance of speed and precision, controlled by a setting called the learning rate. Think of this rate as the pace at which the model adjusts its internal knowledge based on new information. If the pace is too fast, the model overshoots the truth; if it is too slow, it learns nothing. For years, the standard method for managing this pace has been a schedule known as cosine decay. This method starts the model at a moderate speed, gradually slows it down in a smooth curve, and stops it at a precise moment. However, this standard approach has a significant flaw: it requires knowing exactly how long the training will last before the first step is taken. In a world where data arrives continuously and the total amount of information is unknown, this requirement is like trying to drive a car with a map that only shows the destination, not the distance to get there.

Researchers at Harvard University and the Kempner Institute have tackled this problem by developing a new way to train these models that does not need a pre-set end date. Their work, which combines mathematical theory with large-scale experiments, demonstrates that models can be trained effectively without ever knowing when the training will stop. They found that by using a simple, steady pace that naturally slows down over time, and by averaging the model's progress over the journey, they could achieve results just as good as the traditional method. This discovery suggests that the rigid, pre-planned schedules of the past are not necessary for building the most advanced artificial intelligence, opening the door for training systems that can learn continuously as new data becomes available.

The core of the problem lies in the nature of modern data. Unlike the fixed datasets of the past, today's information streams in endlessly. A model might be trained for a week, then for a month, or potentially for years, depending on how much data is available. The traditional cosine decay schedule is "horizon-dependent," meaning it must be tuned specifically for a known duration. If a researcher tunes a schedule for a one-month run but the project extends to three months, the model's performance suffers because the learning rate was designed to slow down too early. The researchers set out to find a "horizon-free" alternative, a method that performs well regardless of whether the training stops tomorrow or next year. They proposed that instead of a complex curve, a simple schedule that decays slowly over time, combined with a technique called weight averaging, could solve this issue. Weight averaging is a process where the final model is not just the last version created, but a blend of many versions from throughout the training process, smoothing out the noise and stabilizing the result.

To test this idea, the team first turned to mathematical theory. They analyzed the behavior of a simplified model of learning, using linear regression on data that mimics the complex patterns found in language. Their analysis showed that for certain types of data, a learning rate that decreases slowly over time, specifically following a pattern where the rate drops as the square root of time increases, is theoretically optimal. Crucially, they proved that when this specific decay is paired with weight averaging, the method achieves the best possible speed of learning without needing to know the total training time in advance. This theoretical finding provided a strong foundation, suggesting that a simple, steady approach could outperform complex, pre-planned curves in an open-ended setting.

The researchers then moved from theory to practice, training large language models with 150 million and 300 million parameters. They trained these models on a massive dataset of text, pushing them to learn up to 32 times the amount of data typically used for models of their size. This scale is significant because it mirrors the conditions under which the world's most advanced AI systems are built. They compared their new horizon-free schedules against the standard cosine decay schedule, which had been carefully tuned for each specific duration of training. The results were striking. The new methods, which used either a constant learning rate with averaging or a slowly decaying rate with averaging, tracked the performance of the best-tuned cosine schedules almost perfectly. Across the entire range of training, from short runs to the longest possible durations, the horizon-free methods achieved final results that were virtually indistinguishable from the traditional approach.

One of the most compelling aspects of this work is its practicality. The researchers showed that a single set of settings, chosen early in the training process, could be used for the entire duration of a project, even if that project suddenly expanded. They demonstrated that if a model is tuned for a specific intermediate point, such as four times the standard data amount, those same settings would continue to perform well as the training extended to 32 times that amount. This eliminates the need to stop and re-tune the model every time the training horizon changes. Furthermore, they tested a popular alternative method known as warmup-stable-decay, which tries to mimic the best of both worlds by keeping the rate steady for most of the training and then decaying at the end. While this method performed well, it still required knowing when to start the decay phase, making it less flexible than the truly horizon-free approach proposed by the authors.

The study also explored how these methods behave when the training data is very large, a scenario where the noise in the learning process is naturally lower. In these conditions, the researchers found that the learning rate did not need to decay at all. A constant learning rate, combined with weight averaging, was sufficient to achieve top performance. This aligns with their theoretical prediction that when the variance, or noise, in the data is small, the model benefits more from maintaining a steady pace to reduce bias, or error, rather than slowing down. This insight suggests that as computing power grows and models can process larger batches of data, the need for complex decay schedules may diminish entirely, replaced by simpler, more robust strategies.

Ultimately, this research provides a clear path forward for the future of artificial intelligence training. It suggests that the rigid, pre-calculated schedules that have dominated the field for years are not the only way, or perhaps even the best way, to train models in an era of continuous data. By proving that simple, horizon-free schedules combined with weight averaging can match the performance of the most carefully tuned traditional methods, the authors have offered a practical solution for open-ended learning. This approach allows models to adapt to whatever amount of data is available, making the training process more flexible and efficient. The findings indicate that the future of large language model training may not lie in more complex scheduling algorithms, but in simpler, more resilient strategies that can learn continuously without a predetermined end.

Technical Summary: Anytime Pretraining with Horizon-Free Learning-Rate Schedules

Problem Statement

Large language models (LLMs) are increasingly trained in continual or open-ended settings where the total training horizon (the total number of tokens to be processed) is not known in advance. Despite this shift, standard pretraining recipes rely on horizon-dependent learning rate schedules, most notably cosine decay. Cosine decay requires knowing the total training length $N$ beforehand to define the decay curve, making it ill-suited for scenarios where data arrives continuously or training budgets are extended dynamically.

Existing "nearly-anytime" alternatives, such as the Warmup-Stable-Decay (WSD) schedule, reduce this dependency but still require additional overhead (storing checkpoints at specific intervals) and re-tuning or re-decaying when extending training, which introduces computational costs and breaks the "anytime" property. The core challenge is to identify a learning rate schedule that:

Does not depend on the training horizon $N$ .
Achieves competitive performance (tracking the "cosine envelope") at any intermediate checkpoint $T < N$ without prior knowledge of $N$ .

Methodology

Theoretical Analysis

The authors provide a theoretical analysis using overparameterized linear regression on Gaussian data with power-law spectra. This serves as a proxy for LLM pretraining.

Setup: The study considers stochastic gradient descent (SGD) with iterate averaging (weight averaging) on a quadratic loss function. The data covariance matrix $H$ and the target vector $w^*$ follow power-law distributions characterized by a capacity exponent $a$ (decay of eigenvalues $\lambda_k \sim k^{-a}$ ) and a source exponent $b$ (decay of target coordinates $(w^*_k)^2 \sim k^{-b}$ ).
Key Insight: The authors analyze the bias-variance decomposition of the risk. They demonstrate that while constant learning rates with averaging can be optimal in certain regimes ( $b \le a$ ), they often require horizon-dependent scaling for $b > a$ .
Proposed Schedule: The paper derives that a polynomial learning rate schedule of the form $\eta_t = \eta / t^\gamma$ $η_{t} = η / t^{γ}$ combined with iterate averaging (specifically Exponential Moving Average, EMA) can achieve minimax optimal convergence rates without knowing $N$ $N$ .
- For the regime $1 < a < b \le 2a$ , the optimal decay exponent is derived as $\gamma^* = 1 - a/b$ .
- A notable special case is $\gamma = 1/2$ (i.e., $1/\sqrt{t}$ ), which is shown to be anytime-optimal up to logarithmic factors and constants under specific spectral conditions.
Weight Averaging: The analysis highlights that weight averaging is central to achieving these rates. Without averaging, standard SGD with polynomial decay often fails to reach optimal rates in the linear regression setting.

Empirical Evaluation

The theoretical prescriptions are stress-tested on LLM pretraining using the OLMo codebase.

Models: Two models were trained: 150M and 300M parameters.
Scale: Training was conducted up to 32 $\times$ the Chinchilla-optimal token count for the 150M model and 16 $\times$ for the 300M model.
Baselines:
- Cosine Envelope: A "gold standard" baseline constructed by independently tuning a cosine schedule for every power-of-2 horizon (1 $\times$ to 32 $\times$ ) and taking the best validation loss at each point.
- WSD: The Warmup-Stable-Decay schedule, implemented by saving checkpoints at 90% of target horizons and decaying for the final 10%.
Candidates:
- Constant LR + EMA: A fixed learning rate with an EMA window growing proportionally to time ( $f=25$ ).
- $1/\sqrt{t}$ + EMA: A polynomially decaying schedule $\eta_t = \eta \sqrt{\alpha/(t+\alpha)}$ with EMA.
Protocol: The authors evaluate whether a single fixed configuration (hyperparameters chosen once) can track the cosine envelope across all intermediate checkpoints, rather than re-tuning for each horizon.

Key Contributions

Theoretical Derivation of Anytime Schedules: The paper proves that for linear regression with power-law data, a learning rate schedule of the form $1/t^\gamma$ combined with iterate averaging achieves minimax optimal rates without dependence on the training horizon. Specifically, it establishes the optimal $\gamma$ as a function of the source and capacity exponents ( $a$ and $b$ ).
Role of Weight Averaging: The work highlights that weight averaging (model merging) is not just a heuristic for generalization but a theoretical necessity for achieving optimal convergence rates with horizon-free schedules in overparameterized settings.
Empirical Validation at Scale: The authors demonstrate that simple, horizon-free schedules (Constant LR + EMA and $1/\sqrt{t}$ + EMA) can track the performance of a carefully tuned, horizon-dependent cosine envelope across a massive training range (up to 32 $\times$ Chinchilla) using a single hyperparameter configuration.
Prospective Transfer: The study shows that hyperparameters tuned at an intermediate checkpoint (e.g., 4 $\times$ Chinchilla) can successfully transfer to significantly longer training runs (up to 32 $\times$ ) without re-tuning, offering a practical protocol for open-ended training.

Results

Performance vs. Cosine: Across the full training range (1 $\times$ to 32 $\times$ ), the proposed anytime schedules (Constant + EMA and $1/\sqrt{t}$ + EMA) achieve validation losses comparable to the independently tuned cosine envelope. The gap is minimal (often within 0.005 validation loss units).
Single Trajectory Success: Unlike cosine decay, which requires re-tuning for different horizons to avoid suboptimal performance at intermediate steps, a single run of the anytime schedules tracks the "cosine envelope" effectively.
Large Batch Regime: In a large-batch experiment (Batch Size 4096), where gradient noise is suppressed, the constant learning rate with averaging outperformed cosine decay and the $1/\sqrt{t}$ schedule. This aligns with the theoretical prediction that when variance is low, decay is unnecessary, and a larger constant rate accelerates bias reduction.
Comparison with WSD: The WSD schedule performs comparably to cosine and the proposed anytime methods. However, WSD is not strictly horizon-free as it requires saving checkpoints at 90% of the run to initiate the decay phase, whereas the proposed methods require no such intervention.

Significance and Claims

The paper claims that weight averaging combined with simple, horizon-free step sizes offers a practical and effective alternative to cosine learning rate schedules for large language model pretraining.

Practicality: The proposed method eliminates the need to know the total training budget in advance, facilitating truly continual and open-ended training.
Simplicity: The schedules are computationally cheaper than WSD (no checkpointing overhead for decay initiation) and easier to implement than horizon-dependent cosine annealing.
Theoretical Grounding: The empirical success is supported by a rigorous theoretical analysis in the linear regression regime, suggesting that the benefits of these schedules are not merely empirical artifacts but stem from fundamental optimization properties in overparameterized settings.

The authors remain modest, noting that their theoretical bounds match minimax rates only up to constants and logarithmic factors, and their empirical conclusions are currently limited to specific model sizes (150M/300M) and optimizers (AdamW without weight decay). They identify future work in understanding how to automatically select the optimal tuning point (e.g., 4 $\times$ Chinchilla) and extending these findings to larger scales and different optimizers.