🤖 machine learning

Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

This paper establishes that the growth of transformer weight scales during training is predictably governed by a pre-computed, training-free statistic of data predictability (bigram conditional entropy), following a specific power-law relationship that allows for accurate forward prediction of weight magnitude changes across different learning rates and architectures.

Original authors: Tiexin Ding

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tiexin Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of artificial intelligence, the most powerful tools are large language models, often called transformers. These systems learn by reading massive amounts of text and adjusting billions of internal numbers, known as weights, to make better predictions about what comes next. For years, researchers have treated these internal numbers as a black box: they know the numbers change during training, but they have struggled to understand exactly how the nature of the text being read shapes those changes. A key question has remained unanswered: does the predictability of the text itself dictate how much the model's internal numbers grow or shift? If a model reads a highly structured, predictable story versus a chaotic, random jumble of words, does the difference show up in the size of the numbers inside the machine?

A new study by independent researcher Tiexin Ding answers this question with a surprising degree of precision. The researcher discovered that the growth of these internal numbers follows a strict, predictable pattern based entirely on a single property of the text: how easy it is to guess the next word based on the one before it. By analyzing a specific type of text corruption, the study found that the more predictable the text, the more the model's internal numbers expand in a specific, measurable way. This relationship holds true regardless of the learning speed used during training, suggesting a fundamental law that connects the raw data to the machine's internal structure.

The research began with a simple observation about the shape of the numbers inside a trained model. When researchers looked at the size of these numbers across different layers of the model, they found they followed a consistent pattern known as a Weibull distribution. This pattern is defined by two values: a shape factor that stays remarkably stable, and a scale factor that changes as the model learns. The stable shape means the model's internal structure remains consistent, while the changing scale factor acts as a dial that turns up or down depending on the training. The study focused on this scale factor, asking what drives it to grow.

To find the answer, the researcher designed a controlled experiment using a standard text dataset. Instead of training on different books or topics, the researcher kept the text the same but systematically scrambled it. They took a fraction of the text and shuffled the order of the words, creating a spectrum of datasets ranging from perfectly ordered to completely chaotic. Before training any model, they measured a specific statistic for each version of the text: the average difficulty of guessing the next word given the previous one. In the clean text, the next word was easy to predict; in the scrambled text, it was nearly impossible.

The results revealed a clear, mathematical relationship between this predictability and the growth of the model's internal numbers. The study found that the growth of the scale factor was not random but followed a specific curve based on the difference between the text's predictability and a baseline of total randomness. Crucially, this relationship was not just a guess; it was derived from two separate, independently measured facts. First, the growth of the numbers was found to be directly proportional to the amount of structure remaining in the text. Second, the difficulty of predicting the next word in the scrambled text followed a specific, curved saturation pattern as the text became more scrambled. When these two facts were combined, they forced the growth of the numbers to follow a precise, convex curve. This means the shape of the curve was not fitted to the data by chance but was a necessary consequence of how the text structure and the learning process interact.

The study tested this law across different learning speeds, which control how quickly the model updates its numbers. When the researcher adjusted for these speeds, all the different training runs collapsed onto a single, unified line. This collapse proved that the relationship is a fundamental property of the system, not an artifact of a specific setting. The researcher then performed a rigorous test to see if the law could predict the future. Using only the pre-training measurement of text predictability, they predicted how much the internal numbers would grow in a new, unseen version of the scrambled text. The prediction was accurate to within a very small margin of error, confirming that the text's properties alone are enough to forecast the model's internal growth.

However, the study also drew a clear boundary around where this law applies. When the researcher tested the model on computer code, the prediction failed. Code has low predictability, which should theoretically lead to high growth according to the law, but the model's numbers grew much less than expected. The researcher identified that code relies heavily on repetition and templates rather than the rich, contextual mapping found in natural language. This suggests that while predictability is a major driver, a second factor—redundancy—also plays a role. The law works perfectly for text where the next word depends on the context, but it breaks down for text where the next word is predictable simply because it repeats a pattern.

The findings also held up when the researcher looked inside the model at the individual layers. The same relationship between text predictability and number growth appeared in every part of the model, though the strength of the effect varied. The layers responsible for processing the flow of information showed the strongest response, while other parts were less sensitive. This allowed the researcher to map exactly where in the network the text's structure was being written into the numbers. Furthermore, the study tested two different model architectures, and while the specific numbers changed, the underlying shape of the relationship remained the same. This indicates that the law is a robust feature of how these models learn, regardless of the specific design of the machine.

Ultimately, this work provides a clear, forward-looking tool for understanding artificial intelligence. It demonstrates that one can look at a dataset before a model is even trained and accurately predict how the model's internal numbers will evolve. This closes a long-standing loop between the data and the machine, showing that the structure of the text is not just a passive input but an active driver of the model's internal dynamics. While the law has limits, particularly with highly repetitive data like code, it offers a powerful new way to measure the quality and structure of training data, turning a complex, invisible process into something that can be measured, predicted, and understood with simple, concrete tools.

Technical Summary: Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

Problem Statement

While the magnitudes of weights in trained transformers follow a stable Weibull distribution (shape parameter $k \approx 1.2$ ), the scale parameter $\lambda$ grows during training. Previous work established that this growth is driven by a balance of three forces (injection, decay, and alignment) under AdamW optimization. However, the specific property of the training data that determines the magnitude of this growth remains an open question. Existing data-evaluation methods often rely on downstream proxies or heuristics; this paper seeks to establish a direct, forward-predictive link between a pre-training corpus statistic and the resulting weight-scale dynamics.

Methodology

The author employs a controlled experimental design to isolate the effect of local next-token predictability on weight growth:

Corruption Gradient: The study uses the WikiText corpus as a base. A corruption level $\rho \in [0, 1]$ is applied, shuffling a fraction of within-sequence token positions. This creates a family of datasets ranging from clean ( $\rho=0$ ) to fully scrambled ( $\rho=1$ ), varying the integrity of the local next-token mapping while holding the corpus fixed.
Data Statistic ( $D$ ): The primary predictor is the bigram conditional entropy, $D = H(\text{next} | \text{prev})$ , computed from the raw token counts of the first 2.4M tokens before any training occurs.
Weight Readout: The author fits a Weibull distribution to the magnitudes of "transmission-class" weight matrices (attention output projections and feed-forward matrices) at various checkpoints. They track the growth of the squared scale parameter, $\lambda^2 - \lambda_0^2$ , where $\lambda_0$ is the initial scale.
Experimental Scope: Experiments were conducted on Pythia-70m (continued training) and a custom Llama-style 70m model (trained from scratch) across four learning rates ( $10^{-4}$ to $10^{-3}$ ) and multiple seeds.

Key Findings and Results

1. The Conditional Entropy Law

The paper establishes a learning-rate-conditioned law relating weight-scale growth to the "predictability margin" $(H_r - D)$ , where $H_r$ is the saturation value of entropy under matched-budget shuffling:
$\lambda^2 - \lambda_0^2 = C_0(\eta) + C_1(\eta) (H_r - D)^{0.59}$

Convex Exponent: The exponent $0.59$ is not a free parameter fitted to the growth curve. It is derived as the reciprocal ( $1/p$ ) of an independently measured data-side saturation exponent ( $p \approx 1.69$ ), linking the growth law to the non-linear saturation of bigram entropy as structure is destroyed.
Learning Rate Collapse: When the per-learning-rate coefficients $C_0(\eta)$ and $C_1(\eta)$ are removed, 23 runs spanning an order of magnitude in learning rate collapse onto a single curve with unit slope ( $R^2 = 0.941$ ). The coefficients themselves follow power-law scalings with the learning rate $\eta$ .

2. Forward Prediction and Self-Validation

Because $D$ is computed before training, the law acts as a genuine forward predictor.

Within-Family Validation: Using a leave-one-out approach across corruption levels, the model predicts weight growth with a 5.7% relative error.
Cross-Corpus Boundary: The law successfully predicts growth for mapping-rich text (e.g., C4) but over-predicts growth for code. Code has low entropy due to redundancy rather than rich structural mapping, causing it to deviate from the law. This identifies redundancy as a second, distinct data dimension.

3. Granularity and Architecture

Layer-Level Resolution: The law holds at the per-block level. The shape parameter $k$ remains uniform across blocks ( $\approx 1.2$ ), while the scale response ( $\lambda$ ) is heterogeneous, concentrated in feed-forward blocks. The pooled model-level growth is well-approximated by the parameter-count-weighted average of per-block growth.
Architecture Independence: The functional form of the law (convex shape, exponent 0.59) is preserved across Pythia (GELU) and Llama-style (SwiGLU) architectures. However, the coefficients differ: SwiGLU feed-forward blocks are approximately twice as sensitive to data predictability as GELU blocks.

Significance and Claims

The paper claims to provide a controlled bridge between a measurable pre-training data statistic and post-training weight dynamics, offering a "forward predictor" of weight-scale growth without requiring training.

Mechanistic Insight: The convexity of the law is explained not by training dynamics, but by the saturation of the static bigram entropy ruler against retained structure. Training remains linear in retained structure, but the data-side metric compresses at the degenerate end.
Data Dimensions: The work argues that data quality is not a single scalar. It proposes a broader framework $\Phi(D, R, A, H)$ where $D$ (mapping predictability) and $R$ (redundancy) are distinct axes. The failure of the law on code quantifies the boundary where redundancy overrides mapping structure.
Scope Limitations: The author explicitly states the law is a within-corpus result for local mapping predictability, not a universal cross-corpus metric. It is established at a specific model scale (~70M parameters) and training budget, well within the transient growth phase before steady-state saturation.

The contribution lies in connecting a standard data-side statistic (conditional entropy) with a Weibull weight-scale statistic through a derived, constrained law, closing the loop between data structure and weight dynamics.