💬 NLP

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

This paper demonstrates that pretraining data is the primary determinant of loss-to-loss scaling laws in large language models, while factors like model size, architecture, and hyperparameters have minimal impact, suggesting that data curation is more critical than architectural choices for optimizing downstream performance.

Original authors: Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, Wieland Brendel

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, Wieland Brendel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect loaf of bread. For years, scientists have been obsessed with the recipe: "If you use 100 grams of flour and 500 grams of water, you get a specific texture." This is similar to how we currently build Large Language Models (LLMs). We have "scaling laws" that tell us how big the model should be and how much data it needs to learn, based on how much computer power we have.

But there's a catch. Just because a model learns the recipe perfectly doesn't mean it will bake a great cake (perform well on real-world tasks).

This paper, "LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws," asks a simple question: What actually determines how well a model performs on new tasks once it has finished learning?

The authors tested dozens of variables—changing the model's brain structure, the size of the model, the tools used to read the text, and even the math used to teach it. They found one massive, surprising truth: The data is the only thing that really matters.

Here is the breakdown using simple analogies:

1. The "Loss-to-Loss" Line

Imagine you are training a student. You give them a practice test (Training Loss) and then a final exam (Test Loss).

The Discovery: There is a very predictable, straight line connecting how well they do on the practice test to how well they do on the final exam.
The Analogy: If a student scores a 90% on practice, they will almost certainly score a 90% on the final, regardless of whether they are wearing a red shirt or a blue shirt. The paper calls this a "shifted power law." It's a reliable rule that holds true across almost everything.

2. The "Big Three" Factors (And Why They Don't Matter)

The researchers played "Mad Libs" with the training process, swapping out different ingredients to see what changed the final result. They found that three major categories had almost zero effect on that predictable line:

The Architecture (The Brain Structure): They compared Llama (a Transformer, like a standard human brain) with Mamba (a State-Space model, like a completely different type of alien brain).
- The Result: If you feed both brains the exact same textbook, they end up on the exact same performance line. It doesn't matter if the brain is shaped like a Transformer or a Mamba; if the data is the same, the outcome is the same.
The Tokenizer (The Dictionary): This is the tool that breaks sentences into words or chunks. They tried different dictionaries (some with 128,000 words, some with 50,000).
- The Result: Changing the dictionary was like changing the font on the textbook. It didn't change how well the student learned the material.
The Settings (The Study Habits): They changed the size of the model (small vs. big), how long the sentences were (context length), and the math used to correct mistakes (optimizers).
- The Result: Whether the student studied for 10 minutes or 10 hours, or used a red pen or a blue pen, the relationship between practice and final scores remained the same.

3. The "One Thing" That Matters: The Data

While the brain structure, the dictionary, and the study habits didn't matter, the textbook itself changed everything.

The Analogy: Imagine two students. Student A reads a textbook about Cooking. Student B reads a textbook about Astrophysics.
- Even if Student A has a super-computer brain and Student B has a simple calculator brain, Student A will be great at cooking and terrible at astrophysics. Student B will be the opposite.
- The paper found that changing the pretraining data (the textbook) shifts the entire performance line. If you train on high-quality educational data (like "FineWeb-Edu"), the model performs better on real-world tasks. If you train on messy data, it performs worse.

The Takeaway for Practitioners

The paper concludes with a clear message for anyone building AI:

Stop obsessing over the architecture and start obsessing over the data.

If you want a model that performs well on real-world tasks, you don't need to invent a new type of brain or tweak the math settings. You need to curate a better dataset.

The Data is the driver. It determines the destination.
The Architecture and Settings are just the car. You can swap the car for a Ferrari or a Honda to make the ride more efficient or cheaper, but if you put the wrong map (data) in the GPS, you'll still end up in the wrong place.

In short: Data determines the destination; everything else just determines how efficiently you get there.

Technical Summary: LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

Problem Statement

Scaling laws have traditionally guided Large Language Model (LLM) development by estimating the optimal balance between model size, token count, and compute budget (measured in FLOPs) to minimize training or validation loss. However, these "compute-to-loss" laws do not reliably predict downstream task performance. While recent work has introduced "loss-to-loss" scaling laws—relationships between training/validation loss and downstream test loss—to better understand generalization, existing studies are largely confined to single architectural setups or training configurations. Consequently, it remains unclear which factors (e.g., architecture, tokenizer, model size, optimization settings) causally influence the shape and transferability of these loss-to-loss scaling trends.

Methodology

The authors conduct a large-scale causal analysis involving over 6,000 model checkpoints to investigate the factors driving loss-to-loss scaling. Their approach extends previous work (Brandfonbrener et al., 2024; Du et al., 2025) by systematically intervening on multiple variables across diverse architectures and training setups.

Experimental Setup:

Architectures: The study compares Transformer-based models (Llama-3, GPT variants) against State-Space Models (Mamba, Mamba2).
Datasets: Models are trained on various pretraining corpora, including FineWeb-Edu, C4, The Pile (original, deduplicated, and uncopyrighted versions), and RefinedWeb.
Tokenizers: Comparisons are made between tiktoken (128k vocab) and gpt2/gpt-neox (approx. 50k vocab) tokenizers.
Interventions: The authors perform ablation studies on:
- Pretraining data distribution.
- Tokenizer choice.
- Model architecture (Transformer vs. State-Space).
- Model size (varying depth and width).
- Context length (1024, 2048, 3076 tokens).
- Optimization settings (Adam vs. AdamW, learning rates, weight decay, and learning rate schedules like Cosine and WSD).
Evaluation: Performance is measured using loss on downstream tasks (e.g., HellaSwag, ARC, MMLU) and validation sets. To ensure fair comparison across tokenizers, loss is normalized to bits-per-byte (BPB).
Modeling: The authors fit shifted power laws to the data, following the formulation:
$L_y \approx K \cdot (L_x - E_{x|p})^\kappa + E_{y|p}$
where $L_x$ and $L_y$ are losses on different datasets, and $E$ represents irreducible error.

Key Contributions and Results

1. Loss-to-Loss Scaling Follows Shifted Power Laws

Across all architectures, datasets, and training configurations, the relationship between training/validation loss and downstream test loss consistently follows a shifted power law. This holds true even for datasets where the model never achieves high accuracy (e.g., Mamba on ARC-Challenge), confirming the robustness of loss-to-loss scaling as a metric for generalization behavior.

2. Pretraining Data is the Dominant Factor

The most significant finding is that the pretraining data distribution is the primary determinant of the loss-to-loss scaling trend.

Changing the pretraining dataset (e.g., from FineWeb-Edu to C4 or The Pile) results in substantial shifts in the scaling curve.
Even subtle changes, such as deduplicating The Pile, alter the underlying distribution enough to shift the scaling law.
This implies that the "inductive bias" governing generalization is largely encoded in the data distribution rather than the model architecture.

3. Architecture, Tokenizer, and Hyperparameters Have Negligible Impact

In contrast to the data, other design choices have minimal to no effect on the scaling trend:

Architecture: Models with fundamentally different inductive biases, such as Transformers (Llama) and State-Space Models (Mamba), exhibit nearly identical loss-to-loss scaling curves when trained on the same data and tokenizer.
Tokenizer: Changing the tokenizer (e.g., tiktoken vs. gpt2) results in only minor variations in the scaling curves, provided loss is normalized by bytes.
Model Size & Context Length: Varying parameter counts (from ~60M to ~2.8B) and context lengths (1024 to 3076) does not meaningfully alter the scaling relationship.
Optimization: Changes in optimizers (Adam/AdamW), learning rates, weight decay, and schedules do not affect the scaling coefficients.

4. Point-wise Generalization

The study demonstrates that if two distinct training setups (e.g., different architectures or sizes) achieve similar training losses on the same pretraining data, they will exhibit closely matched downstream test performance. This suggests that current LLM architectures may converge to similar solutions given the same data, effectively sharing similar inductive biases regarding generalization.

Significance and Claims

The paper argues that the field's focus on architectural innovation for improving generalization may be misplaced. Instead, the authors claim that data curation is the primary driver of downstream performance and robustness.

For Practitioners: The findings suggest that practitioners should prioritize the careful curation of pretraining datasets to optimize downstream performance. Once a high-quality dataset is selected, architectures and training settings can be freely optimized for training efficiency (e.g., choosing faster architectures like Mamba or adjusting hyperparameters) without fear of negatively impacting downstream generalization.
Theoretical Implication: The results challenge the notion that different architectures encode distinct inductive biases regarding generalization. Instead, they suggest that the data distribution dictates the generalization behavior, while architectures primarily serve as efficient function approximators for that data.

The authors maintain a modest stance, acknowledging that their conclusions are based on the specific settings explored (standard loss functions, conventional training). They note that innovative training paradigms (e.g., structured growth or different loss functions) might still induce new scaling behaviors, but within the bounds of current best practices, data remains the decisive factor.