🤖 machine learning

Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

This paper demonstrates that the widely used Chinchilla Approach 2 introduces systematic biases in compute-optimal allocation estimates due to parabolic approximation errors, and proposes Chinchilla Approach 3—enhanced with Variable Projection—as a numerically stable, data-efficient, and unbiased alternative that eliminates these errors while remaining practical to implement.

Original authors: Eric Czech, Zhiwei Xu, Yael Elmatad, Yixin Wang, William Held

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eric Czech, Zhiwei Xu, Yael Elmatad, Yixin Wang, William Held

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to bake the perfect cake. You have a fixed amount of money (your compute budget) to spend on two ingredients: flour (model size/parameters) and eggs (data tokens).

Your goal is to find the perfect ratio of flour to eggs so the cake tastes the best (lowest loss) without wasting a single dollar.

For a few years, the AI world has been using a recipe called "Chinchilla Approach 2" to figure out this ratio. It's popular because it's simple and fast. The paper you shared argues that this popular recipe has a hidden flaw: it systematically overestimates how much flour you need and underestimates how many eggs you need.

Here is the breakdown of why this happens, what it costs, and how to fix it, using simple analogies.

1. The Flawed Recipe: "The Parabola Trap"

The Old Way (Approach 2):
Imagine you want to find the lowest point in a valley (the perfect cake ratio). The old method says: "Let's just look at a small patch of the ground around where we think the bottom is, draw a smooth curve (a parabola) through it, and assume the bottom of that curve is the true bottom."

The Problem:
The actual valley isn't a perfect, symmetrical bowl. It's lopsided.

The Asymmetry: In some AI models (especially those that handle images and sound), the valley is tilted. The "parabola" you draw doesn't sit right in the middle; it slides slightly to the side.
The Drift: Sometimes, the chefs (researchers) don't know exactly where the bottom is, so they guess. If their guess is off-center, the little patch of ground they measure is skewed.

The Result:
Because the valley is lopsided and the measurement patch is often off-center, the "parabola" points to the wrong spot.

The Consequence: The recipe tells you to buy too much flour and too few eggs.
The Cost: In the real world, this means companies like Meta (who built Llama 3) might have spent $1.4 million on extra computer power they didn't actually need. For more complex models (multimodal), this waste could be even higher.

2. Why the "Simple" Method Fails

The paper identifies three main reasons why this simple parabola trick fails:

The Shape of the Valley (Asymmetry): If the loss surface (the valley) is perfectly symmetrical, the parabola works. But most real AI models have "lopsided" valleys. The parabola approximation breaks down, shifting the estimated "best point" away from the truth.
The Grid Width: If you measure a very wide area to draw your parabola, the curve gets distorted more. It's like trying to draw a straight line on a curved road; the wider the road, the more your line misses the mark.
The Drifting Compass: Researchers often don't know the true center. They start their measurements at a guess. If their guess drifts as they get more powerful computers, the error compounds, leading to wildly wrong predictions for future, massive models.

3. The Solution: "The Variable Projection" (Approach 3 & VPNLS)

The paper suggests we stop guessing with parabolas and start measuring the whole valley directly.

The Old Direct Method (Approach 3):
This is like trying to find the bottom of the valley by testing every single combination of flour and eggs simultaneously. It's accurate but very hard. It's like trying to solve a Rubik's cube while juggling; the math gets messy, the computer gets confused, and it often gets stuck in a "local minimum" (a small dip that looks like the bottom but isn't).

The New Smart Method (VPNLS):
The authors propose a clever trick called Variable Projection.

The Analogy: Imagine the valley has two types of knobs: Shape Knobs (how steep the sides are) and Height Knobs (how high the whole valley sits).
The Trick: The math shows that if you fix the Shape Knobs, the Height Knobs become easy to calculate instantly (like solving a simple linear equation).
How it works: Instead of juggling all 5 knobs at once, the computer only searches for the 2 "Shape Knobs." For every guess of the shape, it instantly calculates the perfect height.
The Benefit: This turns a messy, 5-dimensional puzzle into a clean, 2-dimensional one. It's stable, fast, and guarantees you find the true bottom of the valley without the bias of the parabola.

4. The Takeaway for the Real World

The Bias is Real: The "Chinchilla Approach 2" isn't just a little bit off; it has a built-in bias that gets worse as models get more complex (like multimodal AI).
The Cost is High: For a model the size of Llama 3, this bias wasted about 6.5% of the training budget. That's millions of dollars and weeks of compute time.
The Fix is Ready: We don't need to throw out the Chinchilla model. We just need to use a better way to fit the data (VPNLS). It's just as easy to use but gives the correct answer.

In short: The AI world has been using a ruler that is slightly bent to measure the perfect cake recipe. This paper shows us how to straighten the ruler (or use a laser measure) so we stop wasting money on too much flour and too few eggs.

1. Problem Statement

The paper addresses a critical flaw in Chinchilla Approach 2, the most widely adopted method for estimating neural scaling laws. Approach 2 simplifies the complex loss surface by fitting a parabola to "IsoFLOP" curves (constant compute contours) in log-space to estimate optimal parameter ( $N$ ) and token ( $D$ ) allocations.

The authors argue that this parabolic approximation introduces systematic, structural biases that persist even on noise-free data. These biases lead to suboptimal compute allocation, resulting in wasted computational resources (Deadweight Compute Loss). The primary sources of error identified are:

Loss Surface Asymmetry: When the scaling exponents for parameters ( $\alpha$ ) and tokens ( $\beta$ ) are unequal ( $\alpha \neq \beta$ ), the true loss curve is not a perfect parabola. Fitting a parabola shifts the estimated minimum (vertex), biasing the intercept of the scaling law.
Off-Center Sampling: In practice, IsoFLOP grids are rarely centered exactly on the true optimum. If the sampling grid is shifted or "drifts" across compute budgets, it introduces errors in both the intercept and the scaling exponents.
Grid Width: Wider sampling grids amplify the mismatch between the true loss curve and its parabolic approximation.

The paper quantifies these errors on frontier models like Llama 3, estimating that Approach 2 leads to a 6.5% underallocation of parameters (or overallocation of tokens), translating to approximately $1.4 million in unnecessary compute costs for a single training run.

2. Methodology

The authors employ a multi-faceted approach to diagnose and solve these issues:

Synthetic Simulations: They generate noise-free IsoFLOP data from known loss surfaces (symmetric, Chinchilla, and highly asymmetric) to isolate specific error modes. They test various sampling grid widths (from $\pm2\times$ to $\pm16\times$ the optimum) and sampling center offsets (constant and drifting).
Analytical Derivation: They derive a closed-form expression for the vertex shift ( $\delta_w$ ) caused by the parabolic approximation on asymmetric surfaces, proving that the error depends on the asymmetry ratio ( $|\alpha - \beta|$ ) and grid width, but is independent of the compute budget.
Empirical Validation: They analyze published IsoFLOP data from Llama 3, Chinchilla, and DeepSeek. They digitize data points and apply a progressive Quality Control (QC) pipeline to filter out off-center and weak-curvature points, demonstrating that filtering reduces the bias significantly.
Proposed Solution (VPNLS): The authors propose Variable Projection with Non-negative Least Squares (VPNLS). This method exploits the "partially linear" structure of the Chinchilla loss function:
- The loss surface is $L(N, D) = E + A N^{-\alpha} + B D^{-\beta}$ .
- For fixed $\alpha$ and $\beta$ , the parameters $E, A, B$ are linear and can be solved exactly via Least Squares.
- The optimization is reduced from a 5-dimensional nonlinear problem to a 2-dimensional search over $(\alpha, \beta)$ .
- At each candidate $(\alpha, \beta)$ , the linear parameters are solved analytically, ensuring numerical stability and eliminating the ill-conditioning found in direct 5D optimization.

3. Key Contributions

A. Identification of Systematic Biases

The paper demonstrates that Approach 2 is not unbiased even under ideal conditions if the loss surface is asymmetric.

Asymmetry Bias: On asymmetric surfaces, Approach 2 correctly recovers the scaling exponents but produces a biased intercept. This leads to incorrect absolute allocation predictions (e.g., predicting 60% more tokens than optimal for Llama 3).
Drift Bias: If sampling centers drift across compute budgets (a common real-world scenario), Approach 2 also biases the exponents themselves, not just the intercept.

B. Quantification of Economic Impact

Using Llama 3 data ( $3.8 \times 10^{25}$ FLOPs), the authors calculate the Deadweight Compute Loss (DCL):

Llama 3: 6.5% DCL ( $\approx$ $1.4M).
Multimodal Models: For models with higher asymmetry (e.g., SODA, Sparse-NMM), the DCL rises to 8–10% ( $1.7M–$ 2.1M).
Conclusion: The bias is not negligible; it represents a significant waste of resources at frontier scales.

C. The VPNLS Algorithm

The authors introduce VPNLS as a robust alternative to both Approach 2 and standard Approach 3 (direct nonlinear fitting).

Stability: By reducing the search space to 2D, VPNLS avoids the local minima and numerical instability (ill-conditioned Hessians) that plague 5D optimizers like L-BFGS.
Efficiency: It allows for dense grid searches or exhaustive searches over the exponents without the computational cost of a full 5D optimization.
Accuracy: In simulations with noise, VPNLS achieves lower maximum errors and comparable geometric mean errors to the best-tuned Approach 3 implementations, while being structurally simpler.

4. Results

Symmetric Surfaces: Approach 2 works perfectly on symmetric surfaces ( $\alpha = \beta$ ) with centered sampling.
Asymmetric Surfaces: Approach 2 fails to recover the correct intercept. For a highly asymmetric surface ( $\alpha/\beta = 3$ ), the token prediction error reaches -23% with wide grids.
Real-World Data: Analysis of Llama 3 data shows that Approach 2 overestimates the optimal token count by ~60% relative to a properly fitted Approach 3 surface.
Quality Control: Applying a QC pipeline to Llama 3 data (removing off-center points) reduces the DCL from 6.5% to near zero, confirming that the bias stems from sampling design and surface asymmetry.
Method Comparison:
- Approach 2: High structural bias; large maximum errors.
- Naive Approach 3: Unreliable without careful initialization and reparameterization.
- Standard Approach 3 (LSE + Log-Loss): Good accuracy but can be numerically unstable or misspecified for additive Gaussian noise.
- VPNLS: Matches the accuracy of the best Approach 3 variants but with superior numerical stability and a simpler optimization landscape. It reduces the condition number of the Hessian by a factor of $\approx 10^{10}$ .

5. Significance

Resource Optimization: For AI labs training frontier models, switching from Approach 2 to a bias-corrected method (like VPNLS or a carefully tuned Approach 3) can save millions of dollars and weeks of compute time by ensuring the model is trained at the true compute-optimal point.
Methodological Shift: The paper challenges the "simplicity" argument for Approach 2. While Approach 2 avoids nonlinear optimization, it introduces uncorrectable structural errors. The authors argue that Variable Projection offers the best of both worlds: the computational shortcut of separating linear/nonlinear terms without the parabolic approximation error.
Scalability for Future Models: As scaling laws become more complex (incorporating data quality, MoE sparsity, etc.), the linear structure of these extensions can be exploited by VPNLS, making it a more scalable foundation for future research than direct 5D (or higher) optimization.
Reproducibility: The paper highlights that many published scaling law studies omit critical optimization details (initialization, reparameterization), leading to unreliable results. VPNLS provides a more robust, "plug-and-play" alternative.

In summary, the paper provides a rigorous mathematical and empirical case that Chinchilla Approach 2 is fundamentally biased for asymmetric loss surfaces and off-center sampling, leading to significant economic waste. It proposes VPNLS as a superior, stable, and accurate alternative for fitting neural scaling laws.