💬 NLP

Measuring Maximum Activations in Open Large Language Models

This paper presents a comprehensive measurement of maximum activation magnitudes across 27 modern open LLM checkpoints, revealing that peak values vary by nearly four orders of magnitude depending on model family and architecture rather than parameter count, with MoE models showing significantly lower peaks than dense counterparts, thereby establishing maximum activation as a critical, non-monotonic property that must be reported to ensure stable low-bit quantization.

Original authors: Luxuan Chen, Han Tian, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Dawei Yin

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Luxuan Chen, Han Tian, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Dawei Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a giant, digital library where books are read by a very fast, but somewhat fragile, robot. This robot (the AI model) reads words one by one and passes them through a series of rooms (layers) to understand the story. Inside these rooms, the robot generates "excitement levels" (activations) for every word it processes.

Most of the time, these excitement levels are calm and manageable. But sometimes, for a split second, the robot gets extremely excited about a specific word, creating a massive spike in energy.

This paper is like a safety inspection report for 27 different versions of these robots, ranging from small to huge, and from different manufacturers (like Qwen, Gemma, and GPT-OSS). The researchers wanted to answer a simple but critical question: How high can these energy spikes get, and does the size of the robot determine the height of the spike?

Here is what they found, using simple analogies:

1. The "Volume Knob" Problem

Think of the robot's internal energy as a volume knob. To save space and make the robot run faster on cheap hardware, engineers often try to turn the volume down to a low setting (quantization).

The Problem: If the robot suddenly screams at a volume of 1,000,000, but you set your volume knob to only handle up to 10,000, that scream gets "clipped" or distorted. The robot forgets the nuance of that word, and the whole story gets garbled.
The Finding: The researchers measured the loudest scream (the maximum activation) across all these robots. They found that the "loudness" varies wildly. Some robots (like Qwen3.5) are like a whisper, while others (like Gemma3) are like a jet engine taking off.

2. Bigger Doesn't Always Mean Louder

You might think a bigger robot (more parameters) would naturally get louder.

The Analogy: It's like assuming a bigger car always has a louder engine.
The Reality: The researchers found this isn't true. A small robot from one family might scream much louder than a giant robot from a different family. The "loudness" depends more on who built it (the architecture) and how they trained it (the recipe) than just how big it is.
- Example: A Gemma3 robot was found to be roughly 700,000 times louder than a Qwen3.5 robot of a similar size.

3. The "MoE" Shortcut

Some robots use a special trick called "Mixture of Experts" (MoE). Imagine a library where, instead of one giant librarian reading every book, there are 100 small specialists, and only the right 2 or 3 are called in for each book.

The Finding: These "MoE" robots were found to be much quieter. Their loudest screams were 14 to 23 times lower than their "dense" (non-specialist) cousins of the same size. It seems using specialists keeps the energy more contained.

4. Where Does the Scream Happen?

The researchers tracked exactly where in the robot's brain these spikes happened.

The Finding: In almost every case (22 out of 24 robots), the loudest scream happened in the main hallway (the residual stream) where the information flows from one room to the next. It wasn't usually in the side rooms (like the attention or math rooms).
The Pattern: Some robots spike early and stay loud (like a jump-and-hold), while others build up slowly and get loudest at the very end (like a gradual climb).

5. Training Changes the Volume

They also looked at robots at different stages of training, like a student learning a new skill.

The Finding: As the robot trains longer (seeing more books), it tends to get slightly louder. It's not a huge jump, but the "volume" does creep up as the robot gets smarter.

The Bottom Line for Builders

The paper concludes that you cannot guess how loud a robot will be just by looking at its size.

The Takeaway: Before you try to shrink these robots to run on small devices (like phones), you must measure their loudest scream first. If you don't, you might set your volume knob too low, and the robot will start hallucinating or making mistakes because it can't handle the sudden spikes in energy.

The authors have released their measuring tools so that anyone building these robots can check this "loudness" before they try to compress them, ensuring the robot stays stable and accurate.

Technical Summary: Measuring Maximum Activations in Open Large Language Models

Problem Statement

The dynamic range of activations in Large Language Models (LLMs) is a critical constraint for low-bit quantization, activation scaling, and stable inference. Prior research has characterized "outlier features" and "massive activations" primarily on pre-2024 LLaMA-style models (e.g., OPT, BLOOM). However, the deployment landscape has shifted significantly with the release of diverse post-LLaMA open models (e.g., Qwen3.5, Gemma3, GPT-OSS) that vary in normalization stacks, gated MLPs, Mixture-of-Experts (MoE) routing, multimodal adaptation, and training stages.

Existing literature treats massive activations either as a binary categorical phenomenon (existence of outliers) or as a specific obstacle for quantization mitigation techniques (e.g., SmoothQuant, AWQ). There is a lack of systematic, unified measurement regarding:

How large activations can become in modern open LLMs.
How activation magnitude varies across different families, generations, architectures (dense vs. MoE), and training stages.
Whether parameter count alone serves as a reliable proxy for activation range.

Methodology

The authors propose a unified empirical survey to measure the global maximum activation magnitude, defined as $M = \max |a|$ , across a diverse set of models.

1. Data and Corpus:

A shared evaluation corpus of 5,000 multi-domain text samples (Math, Code, English Web, Knowledge/QA, Chinese, Low-resource languages, Mixed Web).
Sequence lengths are bucketed (256 to 4096 tokens), with a heavy skew toward long contexts (93% at 4096 tokens).
Tokenization: The same raw text is re-tokenized for each model family to avoid tokenizer mismatch artifacts.

2. Model Suite:

27 Checkpoints from 8 open families: Qwen2.5, Qwen2.5-VL, Qwen3, Qwen3.5, Gemma2, Gemma3, Ling, and GPT-OSS.
Includes dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.
Specific comparisons are made for matched scales (e.g., MoE vs. Dense, Base vs. Instruct).

3. Measurement Protocol:

Forward-Only Inference: Frozen weights; no parameter updates.
Instrumentation: PyTorch forward hooks collect activation statistics from six component classes:
- Embedding outputs
- Layerwise hidden states (post-residual)
- Attention outputs
- MLP or MoE outputs
- SwiGLU gate pre-activations
- Final LayerNorm outputs
Statistics: For each component, the pipeline records mean, std, RMS, mean absolute value, max/min, and absolute-value quantiles.
Stability Check: Subsampling experiments (1k and 2k samples repeated 5 times) confirm that the reported global maxima are statistically robust and not artifacts of single samples.

4. Analysis Framework:

The study moves from a binary "massive activation" criterion (based on Sun et al. [29]: $|x_i| > 100$ and local ratio $> 1000\times$ ) to a continuous metric $M$ .
Comparisons are conducted across: within-family scaling, cross-family magnitude, generational evolution, MoE vs. Dense, Vision-Language vs. Text-only, Base vs. Instruct, and training-stage evolution.
A lightweight INT-8 sanity check correlates measured maxima with reconstruction error (SQNR) under different scaling strategies.

Key Contributions

Largest Cross-Family Survey: The first unified-protocol measurement of global maximum activations across 24 post-LLaMA checkpoints from 8 families, moving beyond the LLaMA-derivative monoculture of prior work.
Continuous Magnitude Reformulation: Replaces the binary outlier flag with the continuous statistic $M = \max |a|$ . The authors demonstrate that the binary criterion and continuous magnitude can disagree; some models failing the binary criterion are easier to quantize, while some passing it are harder.
Matched-Design Comparisons: Provides observational decompositions of how scale, family, generation, architecture (MoE/Dense), modality (VL/Text), and training progress (Base/Instruct/Stage) affect activation peaks.
Open Pipeline: Releases hook-based measurement code and per-checkpoint statistics to support reproducibility.

Key Results

1. Magnitude Variability and Scaling

Orders of Magnitude: Global maxima span nearly four orders of magnitude ( $10^2$ $1 0^{2}$ to $10^5$ $1 0^{5}$ ) at comparable parameter counts.
- Low Range: Qwen3.5 and MoE checkpoints often fall in the $10^2$ – $10^3$ range.
- High Range: Gemma3-27B-it reaches $\sim 7 \times 10^5$ .
Non-Monotonic Scaling: Cross-family and cross-generation comparisons break simple monotonic scaling. For example, Qwen3.5 activations are significantly suppressed compared to Qwen3, while Gemma3 shows a sharp increase over Gemma2. Parameter count alone is an unreliable proxy for activation range.

2. Architectural and Training Effects

MoE vs. Dense: MoE checkpoints exhibit 14.0–23.4× lower peak magnitudes than matched-scale dense counterparts (e.g., Qwen3-30B-A3B vs. Qwen3-32B).
Residual Stream Dominance: The residual stream (layerwise hidden states) carries the global maximum in 22 out of 24 main checkpoints. GPT-OSS-20B is an exception where the MLP output carries the peak; Qwen3.5-0.8B peaks at the final LayerNorm.
Instruction Tuning (SFT): SFT primarily compresses late-layer peaks (e.g., Qwen2.5-32B Instruct shows a 1.4× reduction in global peak compared to Base) without reorganizing mid-layer structures.
Training Progress: In the Ling-mini series, increasing training tokens (5T to 20T) leads to a monotonic increase in global maximum activation (1.34× increase).

3. Layerwise Patterns

Two distinct layerwise trajectories are observed:

Jump-and-Plateau: Activation magnitude rises sharply in early/middle layers and remains high (e.g., Qwen2.5, GPT-OSS).
Gradual-Accumulation: Magnitude increases smoothly with depth, peaking in later layers (e.g., Qwen3.5, Gemma).

4. Quantization Implications

A lightweight INT-8 sanity check reveals that higher measured maxima ( $M$ ) correlate with substantially lower Signal-to-Quantization-Noise Ratio (SQNR) when using standard per-tensor scaling (max-abs or 99.9% clipping).
Models with low $M$ (e.g., Qwen3.5-0.8B) maintain high SQNR (~29 dB), while high- $M$ models drop to ~10–14 dB or lower under clipping.

Significance and Claims

The paper concludes that maximum activation magnitude is a model property tied to family, architecture, and training stage, not a simple byproduct of size.

Deployment Relevance: $M$ should be measured and reported alongside any open-weight release as part of a deployment-oriented model card. It directly informs the selection of activation scales and quantization strategies.
Complementarity: This work is orthogonal to mechanistic studies (which explain why outliers exist) and quantization mitigation papers (which transform away the upper bound). This paper provides the empirical baseline of how large the bounds are across the current ecosystem.
Limitations: The authors explicitly state they do not claim causal mechanisms for the observed differences. The study is observational, limited to open models, and uses a fixed corpus (up to 4k tokens). The INT-8 check is a sanity check, not a full end-to-end quantization evaluation.

The authors recommend that future deployment pipelines treat $M$ as a critical prior for quantization risk, acknowledging that family identity and generation are at least as important as parameter count in determining activation dynamic range.