🤖 machine learning

A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions

This paper introduces a two-parameter Weibull distribution framework for diagnosing transformer weight magnitudes, revealing that feed-forward and output projection layers consistently converge to a specific shape parameter (k ≈ 1.20) regardless of architecture, while input projections deviate based on storage mechanisms and the scale parameter (lambda) tracks training progress.

Original authors: Tiexin Ding

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tiexin Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Transformer model (the kind of AI that powers chatbots and code generators) as a massive, complex orchestra. For years, researchers have tried to understand how this orchestra learns by listening to the "sound" of the whole group or looking at the sheet music as a single, blurry sheet.

This paper introduces a new way to listen: a microscope for individual notes. Instead of looking at the whole orchestra, the authors developed a tool to examine the "weight" (the strength) of every single instrument's string. They found that these strings don't just get stronger or weaker randomly; they follow a very specific, predictable pattern that reveals exactly what the AI is doing.

Here is the breakdown of their discovery using simple analogies:

1. The "Ruler" They Invented (The Weibull Distribution)

The authors used a mathematical shape called the Weibull distribution. Think of this as a special ruler with two settings:

The Shape ( $k$ ): This measures the "personality" of the weights. Is the distribution smooth and uniform, or does it have wild, heavy tails (extreme outliers)?
The Scale ( $\lambda$ ): This measures the "volume" or size of the weights.

The Magic Anchor:
When the AI starts training (before it learns anything), its weights are set randomly, like a Gaussian bell curve. The authors proved that if you measure this starting state with their ruler, the "Shape" ( $k$ ) always lands on a specific number: 1.20.

Think of 1.20 as "Zero." It's the baseline. If a part of the AI stays at 1.20, it hasn't changed its fundamental nature. If it moves away from 1.20, something interesting has happened.

2. The Two Types of Musicians

The most exciting finding is that the AI's internal parts naturally split into two distinct groups, like two different sections of an orchestra:

Group A: The "Transmitters" (The Steady Players)

Who they are: The parts that pass information through the model (like the Feed-Forward Network and the Attention Output).
What they do: They act like a reliable highway. They move data from point A to point B without changing the nature of the road.
The Result: Even after training, their "Shape" ( $k$ ) stays incredibly close to the starting 1.20. They are stable, consistent, and don't change their personality.
Analogy: Imagine a delivery truck that drives the same route every day. It might get bigger (carry more cargo), but it's still the same truck.

Group B: The "Selectors" (The Specialized Hunters)

Who they are: The parts that decide what to pay attention to (the Query and Key projections in the attention mechanism).
What they do: They have to be picky. They need to amplify some signals and ignore others to make smart decisions.
The Result: Their "Shape" ( $k$ ) drifts away from 1.20, usually dropping lower (between 0.76 and 1.16). This means they develop "heavy tails"—a few extreme, super-strong connections emerge while the rest stay normal.
Analogy: Imagine a detective who starts with a broad net. As they learn, they stop casting a wide net and instead develop a few incredibly sharp, specialized tools to catch specific clues. Their "personality" changes drastically from the starting point.

3. How Architecture Changes the Game

The paper found that the type of AI architecture changes how much the "Selectors" drift:

Separate Heads (MHA): If the AI has many independent "heads" (like OLMo), the Selectors go wild. Their shape drops the most (to ~0.76). They become very specialized.
Grouped Heads (GQA): If the AI groups its heads together (like LLaMA-3 or Mistral), the Selectors are less extreme. They still drift, but not as far (around 1.10–1.16). It's like having a team of detectives sharing notes; they can't all go off in totally different directions.
Merged Heads (Pythia): If the AI merges them completely, they sit right in the middle, acting as a transition between the two styles.

4. Volume vs. Personality

The authors discovered that the two settings on their ruler tell two different stories:

The Scale ( $\lambda$ ) tells you how far along the training is. As the AI learns, the "volume" of the weights grows. This is like the orchestra getting louder. This growth follows a predictable math rule related to how fast the AI is learning (learning rate) and how much it is being "disciplined" (weight decay).
The Shape ( $k$ ) tells you what the part does. It tells you if a part is a "Transmitter" (stable) or a "Selector" (specialized).

5. The "Super-Weights" (Dragon Kings)

The paper also noticed that while most weights stay normal, a tiny handful of individual weights become massive outliers (called "super-weights").

The Analogy: Imagine a choir where 99% of singers are singing at a normal volume, but one singer suddenly starts screaming at a level 100 times louder than everyone else.
The Finding: These "screamers" appear in every model. However, the authors' "middle-80% rule" (ignoring the loudest and quietest 10% when measuring) allows them to ignore these screamers and still get an accurate reading of the choir's true character. This proves that the "Transmitters" stay stable even while a few "rogue" weights go crazy.

Summary

The paper gives us a new diagnostic tool. By measuring the "Shape" and "Scale" of an AI's weights, we can:

Identify which parts of the AI are stable (Transmitters) and which are learning to specialize (Selectors).
See how the AI's architecture (how it's built) forces these parts to behave differently.
Track the AI's progress by watching the "volume" grow, without getting confused by the few "rogue" weights that scream the loudest.

It turns the black box of AI training into a readable map, showing us exactly where the learning is happening and how the model is structuring its knowledge.

Technical Summary: A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions

Problem and Motivation

Current diagnostic tools for transformer training dynamics operate in orthogonal spaces that fail to capture the granular structure of learned weights. Methods like WeightWatcher and HT-SR analyze eigenvalue spectra, compressing matrix elements into singular value distributions, while AlphaDecay tracks top singular value drift. Massive activation analysis measures activation magnitudes. None of these approaches directly characterize the element-wise distribution of weight magnitudes ( $|W_{ij}|$ ), which the authors argue is the most granular representation of what a model has learned.

The authors posit that aggregate spectral statistics obscure systematic differences between functional components within the same model (e.g., FFN layers vs. attention projections). To address this, the paper proposes applying the Weibull distribution—a two-parameter family from extreme-value theory—as a diagnostic lens for element-wise weight magnitudes.

Methodology

The Weibull Diagnostic Framework

The framework relies on fitting the absolute values of weight elements, $|w|$ , to a Weibull distribution defined by a shape parameter $k$ and a scale parameter $\lambda$ :
$f(x; k, \lambda) = \frac{k}{\lambda} \left(\frac{x}{\lambda}\right)^{k-1} \exp\left(-\left(\frac{x}{\lambda}\right)^k\right)$

1. The Initialization Anchor:
At initialization, transformer weights are i.i.d. Gaussian ( $w \sim \mathcal{N}(0, \sigma_{init}^2)$ ), meaning $|w|$ follows a Half-Normal distribution. By fitting this Half-Normal distribution to a Weibull model using a specific protocol, the authors establish a universal anchor:

Shape ( $k_0$ ): Approximately 1.20. This value is independent of the initialization scale $\sigma_{init}$ and serves as a dimensionless measuring stick. Any deviation from $k_0 \approx 1.20$ in a trained model is attributable to training dynamics.
Scale ( $\lambda_0$ ): Approximately $0.8875 \cdot \sigma_{init}$ .

2. The Fitting Protocol (Middle-80% Trim):
To ensure robustness against outliers and numerical precision artifacts, the authors employ a middle-80% trim protocol. Before fitting, the smallest and largest 10% of $|W_{ij}|$ values are discarded.

Theoretical Justification: In Weibull probability coordinates, the sampling noise variance diverges at the tails ( $p \to 0$ and $p \to 1$ ) and reaches a minimum near $p \approx 0.80$ . The bottom 10% carries ~6.5 $\times$ more noise than the optimal central region.
Effect: Fitting full data systematically underestimates $k$ by 3–7%, often pushing models outside the expected "Transmission" band. The 80% trim isolates the "body" of the distribution, making the metric robust to heavy-tailed outliers (super-weights).

3. Functional Classification:
The framework partitions transformer components into two distinct classes based on their $k$ trajectories:

Transmission Class: FFN modules ( $W_{gate}, W_{up}, W_{down}$ or $W_{FFN\_in}, W_{FFN\_out}$ ) and attention output projections ( $W_o$ ).
Selection Class: Attention input projections ( $W_q, W_k$ ).

Key Findings

The framework was applied to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) with parameter counts ranging from 70M to 14B.

1. The Transmission Band (Stability)

Components in the Transmission Class exhibit remarkable stability in their shape parameter $k$ .

Across all architectures, activation functions (SwiGLU, GeLU), normalization placements (Pre-LN, QK-Norm), and model sizes, the median terminal $k$ for these components falls in the narrow interval $[1.186, 1.204]$ .
The coefficient of variation (CV) across the 12 entries is only 0.51%.
This stability suggests that these components preserve the initialization distributional shape, acting as information conduits rather than selective filters.

2. The Selection Class (Departure)

Components in the Selection Class ( $W_q, W_k$ ) systematically depart from the initialization anchor ( $k_0 \approx 1.20$ ), developing heavier tails (lower $k$ ). The severity of this departure is architecture-dependent:

Separately-stored MHA (OLMo-1, OLMo-2): Shows the deepest drift, with median terminal $k \in [0.76, 0.99]$ .
Grouped-Query Attention (GQA) (LLaMA-3, Mistral, Qwen2.5/3): Shows milder drift, with median terminal $k \in [1.10, 1.16]$ .
Merged $W_{qkv}$ (Pythia): Occupies a transitional zone $k \in [1.05, 1.18]$ , tracking the dimensionless training budget $T/\tau$ monotonically.
Mechanism: The authors attribute this drift to functional necessity (sparse attention), AdamW sign-descent dynamics, and softmax saturation feedback. The architectural constraint of K-head sharing in GQA appears to mechanically limit the specialization freedom available to individual projections, resulting in less severe drift compared to MHA.

3. Scale Parameter ( $\lambda$ ) and Training Progress

While $k$ labels the functional class, the scale parameter $\lambda$ labels training progress.

$\lambda$ grows substantially during training (by factors of 1.4 $\times$ to 7.8 $\times$ ).
Within the Pythia family, the terminal mean $\lambda$ for Transmission components scales with $\sqrt{\eta / \lambda_{wd}}$ (Pearson $r = 0.94$ ), showing directional consistency with the AdamW steady-state scaling analysis of Fan et al. [2025].
$\lambda$ grows uniformly across component types (e.g., $\lambda_O$ and $\lambda_{FFN\_out}$ are highly correlated, $r=0.9967$ ), reflecting coupled dynamics in the residual stream.

4. Super-Weights and Tail Behavior

While the body of the distribution (fitted via 80% trim) remains Weibull-conforming, the authors observe the emergence of isolated "super-weights" (dragon-king outliers) in the extreme tails of trained models.

These outliers appear in every architectural family, with max-to-q99 ratios ranging from ~7 $\times$ to over 100 $\times$ .
The middle-80% protocol is specifically designed to be robust to these outliers, ensuring that the diagnostic $k$ reflects the bulk distribution rather than being dragged by extreme values.

Contributions

A Diagnostic Framework: The authors formalize a method using the Weibull distribution with a principled initialization anchor ( $k_0 \approx 1.20$ ) and a noise-optimal fitting protocol (middle-80% trim). This framework identifies that $W_q$ and $W_k$ systematically depart from the initialization anchor, with the magnitude of departure shaped by storage architecture.
Cross-Family Empirical Evidence: The paper documents a robust "Transmission/Selection" partition across 12 models. It reproduces the narrow FFN transmission band and the architecture-dependent Selection drift (MHA vs. GQA) without tunable parameters.
Open-Source Tools: The authors release npm-weibull-py v0.4, a Python library implementing eight diagnostic functions (F1–F8) for fitting and benchmarking, alongside DATABASE_v9_1, a benchmark database of 12 model entries.

Significance and Limitations

Significance:
The paper claims that element-wise statistics reveal structure that spectral methods cannot. By separating components into Transmission and Selection classes, the framework provides a granular view of training dynamics where different layers and projection matrices evolve along distinct trajectories. The two Weibull parameters carry independent information: $k$ identifies the functional role (Transmission vs. Selection), while $\lambda$ tracks the magnitude of training progress.

Limitations (as stated by the authors):

Causal Mechanisms: While the paper identifies five candidate driving forces for Selection drift (functional necessity, sign-descent, softmax saturation, residual coupling, training budget), it does not provide controlled ablation experiments to isolate the contribution of each. The correlation between MHA/GQA architecture and drift severity is observational.
Cross-Family Scaling: The scaling law $\lambda \propto \sqrt{\eta / \lambda_{wd}}$ is validated within the Pythia family but is only observational across different families due to confounding factors like different initialization recipes and normalization placements.
Downstream Performance: The paper characterizes the statistical structure of weights but does not establish a direct correlation between the $(k, \lambda)$ signature and downstream task performance.
Data Coverage: For Qwen2.5-14B, per-block statistics are limited to the first 27 of 48 layers due to GPU memory constraints, though qualitative claims are expected to hold.

The authors conclude that this framework offers a new, architecture-independent measuring stick for diagnosing transformer training, revealing that the "Selection" components ( $W_q, W_k$ ) are the primary sites of structural change, while "Transmission" components remain statistically stable.