🤖 machine learning

Small-Scale Experiments: Are We There Yet?

This paper argues that the perceived failure of small-scale experiments to validate scaling laws stems from hyperparameter sensitivity rather than model size, demonstrating that with proper tuning and a holistic methodology, small models can reliably predict large-scale outcomes such as the superiority of pre-normalization in transformers.

Original authors: Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, Sanae Lotfi

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, Sanae Lotfi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect loaf of bread, but you only have a tiny, cramped kitchen. You want to know how to bake a giant, city-sized loaf for a festival, but baking that giant loaf would cost a fortune in flour and oven time. So, you decide to test your recipes on tiny, 4-ounce rolls first. This is the dream of "scaling laws" in the world of artificial intelligence: the idea that if you understand how a small AI model learns, you can predict exactly how a massive one will behave without having to spend millions of dollars to build it.

For a long time, scientists hoped this "small-to-large" shortcut would work perfectly. They believed that if they tweaked the recipe for a tiny model, they could just scale up the ingredients and get a giant, perfect model. But lately, things have been messy. When researchers tried to use their small experiments to predict the behavior of huge models, the predictions often failed. It was like the tiny rolls tasted great, but the giant loaves turned out burnt or flat. The big question became: Is the shortcut broken? Do we really have to build the expensive giant loaf every single time to see if it works?

This paper, titled "Small-Scale Experiments: Are We There Yet?", dives into this kitchen mystery. The authors, a team from Meta and New York University, argue that the shortcut isn't broken; we just haven't been looking at the right part of the recipe. They discovered that the problem isn't the size of the model, but how carefully we tune the "knobs and dials" (called hyperparameters) on the small ones.

Think of a small AI model like a very sensitive, high-performance race car engine. If you turn the fuel mixture knob just a tiny bit wrong, the engine sputters and dies. It's incredibly hard to get it running perfectly. A giant AI model, on the other hand, is like a massive, slow-moving steamship. It's much more forgiving; even if you adjust the knobs a bit clumsily, the ship keeps moving forward just fine. The authors found that because small models are so sensitive, researchers often miss the "perfect" setting because they don't try enough different combinations. They might test four or sixteen settings and say, "This is the best we can do," not realizing that the true "perfect" setting is hiding somewhere else in the vast ocean of possibilities.

The paper suggests that if you are willing to do the hard work of testing hundreds of different settings on your tiny models, you can find the perfect recipe. Once you find that perfect setting for the small model, the rules for how it scales up become clear and predictable. They showed this by testing two different ways of building AI brains (called "pre-normalization" and "post-normalization"). In the past, figuring out which one was better took years and massive computers. But by using their new method of intense, small-scale testing, they were able to predict the winner correctly using only tiny models.

The authors also explain why this happens using a cool geometric idea. They say that as a model gets bigger, the "landscape" of possible settings becomes simpler. For a small model, the landscape is a jagged, confusing mountain range with thousands of hidden valleys where the engine might stall. But as the model grows, that landscape smooths out into a gentle, wide valley where it's much easier to find the bottom. This means that while small models are hard to tune, big models are actually easy to tune.

So, the big takeaway is that we don't have to give up on small experiments. We just need to stop skimming the surface with our testing. If we treat small models with the respect they deserve—testing them thoroughly rather than just skimming the surface—we can save a massive amount of money and time. We can finally trust our small-scale experiments to tell us the truth about the giants, proving that the "small-to-large" shortcut is still valid, provided we have the patience to find the right settings first.

Technical Summary: Small-Scale Experiments: Are We There Yet?

Problem Statement

Despite the promise of scaling laws to enable cost-effective model development by extrapolating from small-scale experiments, the field has largely failed to realize this potential. Six years after the initial proposal of scaling laws, researchers often find them unreliable at small scales (starting as low as 4M parameters), leading to the conclusion that sizable models are unavoidable for reliable insights. This has resulted in an exponential increase in the cost of pretraining research, with experimental runs dominating development expenses.

The core issue identified is a "scaling gap": insights from small-scale experiments frequently fail to transfer to production systems. Previous literature has attributed discrepancies in scaling laws (e.g., between Kaplan et al. and Hoffmann et al.) to methodological choices such as parameter counting or learning rate schedules. However, this paper argues that the primary confounding factor is hyperparameter tuning. Small-scale models are highly sensitive to hyperparameters, and the "fully tuned frontier" required to observe scaling laws is often missed because the necessary search space is far more extensive than typical academic or industry runs.

Methodology

The authors propose a holistic, model-centric research methodology that synthesizes three theoretical frameworks to analyze small-scale experiments:

The Noisy Quadratic Limit: Used to diagnose whether hyperparameters are fully tuned. Near the optimum, the loss surface is approximated as quadratic plus additive normal noise. The tail of the validation score distribution from random search converges to a noisy quadratic distribution ( $Q_{min}$ ), characterized by parameters including the best attainable performance ( $\alpha$ ), concentration ( $\beta$ ), effective number of hyperparameters/intrinsic dimension ( $\gamma$ ), and noise ( $\sigma$ ).
Scaling Laws: The standard functional form relating parameters ( $p$ ), data ( $d$ ), and loss ( $L$ ) is analyzed, specifically focusing on the "fully tuned frontier" where loss is minimized for a given compute budget.
Perplexity–Capability Correspondence: Based on recent findings that pretraining loss corresponds to downstream capabilities if the pretraining data is held fixed. This allows researchers to use loss as a proxy for capability without needing to predict specific downstream behaviors.

Experimental Design:

Models: Experiments were conducted on Transformer models ranging from 4M to 268M effective parameters.
Search Strategy: Extensive random searches were performed across scales. Crucially, the authors varied the number of configurations per scale (4, 16, 64, and 256) to demonstrate the necessity of search depth.
Training: Models used a warmup-stable-decay (WSD) schedule. Checkpoints were branched off at various training stages to evaluate both stable and decayed learning rates.
Case Study: The methodology was applied to resolve the long-standing question of normalization layer placement (Pre-Norm vs. Post-Norm) in Transformers, a debate that previously took years and theoretical analysis to settle.

Key Contributions and Results

1. Hyperparameter Tuning is the Critical Ingredient

The paper demonstrates that scaling laws are robust to methodological details (e.g., how parameters are counted, learning rate decay, or tying exponents) but are extremely sensitive to the extent of hyperparameter search.

With only 4 or 16 configurations per scale, scaling laws are invisible or unreliable due to noise from suboptimal configurations.
Only with extensive search (256 configurations) do accurate scaling laws emerge, even at 4M parameters.
The "fully tuned frontier" is the only place where predictable scaling appears; small models are harder to tune, making their laws harder to observe, not absent.

2. The Geometry of the Hyperparameter Loss Surface

The paper explains why small models are harder to tune through the geometry of the loss surface:

Dimensionality Reduction: As model scale (parameters and data) increases, the intrinsic dimension of the hyperparameter loss surface ( $\gamma$ ) decreases.
The "Blessing of Dimensionality": While small models have a high-dimensional, unforgiving loss surface where good configurations are rare, larger models exhibit a lower-dimensional surface where good configurations fill a larger fraction of the space.
Driver: Parameters are the primary driver of this dimensionality reduction, more so than data. As models scale, the effective number of hyperparameters drops toward one, making large models easier to tune with simple rules.

3. The Limits of Pure Extrapolation

While scaling laws exist at small scales, the paper cautions against blind extrapolation.

Statistical Limits: Extrapolating scaling laws far beyond the training data magnifies sampling errors and irreducible noise. Comparing models based on extrapolated loss far from the data is unreliable.
Holistic Approach: A qualitative understanding of the loss surface (e.g., checking if the noisy quadratic limit emerges, verifying sensitivity decreases with scale) is required alongside quantitative extrapolation.

4. Case Study: Pre-Norm vs. Post-Norm

Applying the new methodology to normalization layers:

Tuning Difficulty: Post-Norm architectures were found to be significantly harder to tune than Pre-Norm, exhibiting a smaller asymptotic regime and persistent suboptimal peaks in the score distribution.
Scaling Behavior: Pre-Norm scaled better near the data. When assuming both architectures reach the same final loss, Pre-Norm maintained a scaling advantage.
Result: The small-scale experiments, when rigorously tuned and analyzed holistically, correctly recovered the large-scale consensus that Pre-Norm is superior, a conclusion that previously required years of research.

Significance and Claims

The paper claims that the "scaling gap" is not an inherent limitation of small models but a consequence of insufficient hyperparameter search and a lack of understanding of the loss surface geometry.

Feasibility: Scaling laws do exist at very small scales (as low as 4M parameters), but they only emerge on the fully tuned frontier.
Cost Efficiency: Reaching this frontier requires extensive search, but because small models train quickly (under an hour on a single GPU), this search is computationally affordable compared to running large-scale experiments.
Methodological Shift: The authors advocate for a shift in model-centric research: thoroughly explore the hyperparameter space at small scales to build a qualitative understanding of the loss surface, then use simple rules to transfer these findings to larger scales.
Limitations: The approach is currently limited to model-centric research (improving the model with fixed data). Changing the pretraining data breaks the perplexity–capability correspondence, limiting the ability to predict large-scale outcomes for data-centric changes using small-scale proxies.

In conclusion, with the right tools (noisy quadratic analysis, extensive random search) and a better understanding of hyperparameter sensitivity, small-scale experiments can finally deliver on the promise of scaling laws, offering a path to reduce costs while increasing research rigor.