💬 NLP

A Comparative Analysis of LLM Memorization at Statistical and Internal Levels: Cross-Model Commonalities and Model-Specific Signatures

This study bridges isolated observations across multiple LLM families (Pythia, OpenLLaMa, StarCoder, OLMo) to establish a universal understanding of memorization by revealing shared log-linear scaling and internal decoding patterns while identifying distinct model-specific signatures at both statistical and internal levels.

Original authors: Bowen Chen, Namgi Han, Yusuke Miyao

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bowen Chen, Namgi Han, Yusuke Miyao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) like super-obsessive students who have read the entire internet. These students are so smart they can write essays, code, and poems. But there's a catch: sometimes, instead of understanding the material, they just memorize it word-for-word. This is called "memorization."

This paper is like a detective story where researchers investigate how these students memorize things. They didn't just look at one student; they looked at 20 different students from 6 different "schools" (model families like Pythia, OLMo, StarCoder, etc.) to see if they all study the same way or if every school has its own unique study habits.

Here is the breakdown of their findings, using simple analogies:

1. The "Size vs. Memory" Mystery

The Finding: Generally, the bigger the student (more parameters), the more they memorize. It's a straight line: double the size, and you get a predictable jump in memory.
The Twist: However, not all big students are equal.

Analogy: Imagine two students, "StarCoder" and "OLMo." StarCoder is like a student who only read a library of coding manuals. Because they only read code, they memorized it almost perfectly (high memorization rate). OLMo is like a student who read everything (books, news, code, math). Because their library was so huge, they memorized a smaller percentage of it, even though they are a "bigger" student.
Lesson: Just because a model is huge doesn't mean it memorizes more; it depends on what it was fed.

2. The "Compression" Trick

The Finding: When a model memorizes something, it doesn't need the whole sentence to remember it. It can recall the whole thing from just a tiny hint.
The Analogy: Think of a zip file. If you memorize a long poem, you don't need to recite the first 50 words to remember the rest. You might just need the first two words ("Once upon..."), and your brain instantly fills in the rest.

The researchers found that for some models, you only need half of the original text to trigger the full memory. The model has "compressed" the memory into a tiny, efficient package.

3. What Do They Memorize? (The "Hard" vs. "Soft" Stuff)

The Finding: Models love to memorize structured things (like code, math formulas, or legal contracts) because they follow strict rules. They are bad at memorizing free-flowing things (like a casual conversation or a story) because those are messy.
The Twist: As models get massively bigger, they start getting better at memorizing the "messy" stuff (free text).

Analogy: A small student can easily memorize a recipe (strict rules: 2 cups flour, 1 egg). But a giant student starts to memorize jokes or poems (messy, emotional, no strict rules) because they finally understand the meaning behind the words, not just the pattern.

4. The "Noise" Test (Can They Ignore Distractions?)

The Finding: The researchers tried to "distract" the models by adding random noise (like static on a radio) to their brain while they were trying to recall a memorized fact.

General Brain: The model is good at ignoring the noise and cleaning up the signal (like a good listener ignoring background chatter).
The Memorized Fact: However, the memorized facts were actually more sensitive to the noise than normal facts.
Analogy: Imagine you are trying to remember a phone number you memorized. If someone whispers random numbers in your ear, you might get confused and forget the exact sequence. But if you are just thinking about a concept (like "what is a phone?"), the whisper doesn't bother you as much.
Conclusion: Memorized sequences rely on very specific, fragile pathways in the brain. If you poke those pathways, the memory breaks.

5. The "Brain Scan" (Where does the memory live?)

The Finding: The researchers looked inside the model's "brain" (its layers and attention heads) to see where the memories are stored.

The Common Pattern: All models seem to have a "two-stage" process.
1. Early Layers: The brain grabs the "gist" or the raw data.
2. Late Layers: The brain turns that data into the final answer.
The "Special Agents": There are a few specific "agents" (attention heads) in the brain that are crucial for all types of memorization, no matter the topic.
The School Signature: However, where these agents sit in the brain is different for every school.
- Analogy: Imagine two different companies. Both have a CEO and a Janitor (the universal parts). But in Company A, the CEO sits on the 2nd floor. In Company B, the CEO sits on the 10th floor. The structure of the company is unique to its training. You can tell which "school" a model came from just by looking at where its "memory agents" are sitting.

The Big Takeaway

This paper tells us that while all LLMs are built similarly, how they memorize is a fingerprint of their training.

Universal Truth: Bigger models generally memorize more, and they compress memories tightly.
Unique Signature: The specific way a model memorizes (which parts of the brain it uses, how sensitive it is to noise) is determined by the specific recipe (data and training method) used to create it.

Why does this matter?
If we want to stop AI from leaking private data (memorizing your credit card number), we can't just make the model bigger or smaller. We have to understand the specific "fingerprint" of that model's memory to know exactly where to look and how to fix it.

1. Problem Statement

Memorization (the ability of Large Language Models to regurgitate content from their pre-training corpora) is a fundamental component of intelligence but poses significant risks regarding privacy and copyright. While previous research has analyzed memorization, it suffers from two major limitations:

Data Scarcity: Most studies rely on "sandbox" models or single model series where pre-training data is inaccessible, limiting the generalizability of findings.
Isolated Observations: Findings are often specific to one model family, making it unclear which mechanisms are universal (cross-model commonalities) and which are artifacts of specific training recipes (model-specific signatures).

This study aims to bridge this gap by conducting a large-scale comparative analysis across multiple model families with known, accessible pre-training data to identify universal laws of memorization versus family-specific traits.

2. Methodology

The authors analyzed 20 models across 6 distinct families (Pythia, OpenLLaMA, StarCoder, OLMo1, OLMo2, OLMo3), ranging from 160M to 32B parameters. The pre-training corpora sizes spanned from billions to trillions of tokens.

Key Experimental Components:

Memorization Score ( $M_i$ ): Calculated using greedy decoding on 32 input tokens from the pre-training corpus to see if the model perfectly recalls the subsequent 32 tokens.
Compression Ratio: Measured the minimum number of context tokens required to trigger the full memorized sequence, assessing the "firmness" of the memory.
Noise Perturbation: Injected Gaussian noise into the residual stream at the first layer to test the robustness of memorized sequences versus unmemorized ones.
Logit-Lens Decoding: Probed the internal decoding probabilities of tokens across different layers to understand the emergence of memorization.
Attention Head Ablation: Systematically removed individual attention heads to calculate an "Importance Score" ( $I_M$ ), identifying which heads are critical for generating memorized sequences.
Domain Classification: Categorized memorized sequences into Structural (code, math), Semi-Structural (legal, news), and Free-Text (natural language).

3. Key Contributions

The study provides the first comprehensive mapping of memorization behaviors across diverse model architectures and training data, distinguishing between universal scaling laws and specific architectural signatures.

A. Statistical Level Findings

Log-Linear Scaling: Memorization rates scale log-linearly with model size (parameter count) across all families. However, the absolute rate varies significantly (up to 100x difference) between families, heavily governed by the specific pre-training corpus.
High Compressibility: Memorized sequences are intrinsically highly compressed. For many models (e.g., Pythia, StarCoder), less than 50% of the original context tokens are sufficient to trigger the full memorized output.
Frequency & Domain Shifts:
- Frequency: Memorized sequences generally follow a high-frequency threshold, but larger models lower this threshold, allowing them to memorize lower-frequency tokens.
- Domain: While structural texts (code/math) are easiest to memorize, scaling model size shifts the balance toward memorizing free-text (natural language), suggesting improved semantic understanding aids memorization.
Bipolar Distribution: Memorization scores exhibit a bipolar distribution: most sequences are either fully unmemorized or fully memorized, with few intermediate states.

B. Internal Level Findings

Denoising vs. Sensitivity: LLMs possess a general internal denoising capability (recovering similarity in later layers after early-layer noise injection). However, memorized sequences are significantly more sensitive to noise than unmemorized ones, implying they rely on specific, fragile computational pathways.
Two-Stage Decoding Mechanism:
- Early Layers: Establish core features and routing for exact memorization.
- Late Layers: Activate and translate prepared information into final output probabilities.
- Memorized tokens show a distinct probability advantage in later layers compared to unmemorized tokens.
Head Importance & Structure:
- Shared Heads: A small subset of attention heads is crucial for memorization across almost all domains (>90% of domains).
- Family Signatures: While specific heads are shared, their layer-wise distribution forms a unique "structural fingerprint" for each model family. This distribution is consistent within a family (e.g., all OLMo models) but diverges across families, shaped by the specific training recipe (data + algorithm).

4. Key Results Summary

Feature	Cross-Model Commonality (Universal)	Model-Specific Signature (Family-Dependent)
Scaling	Memorization rate scales log-linearly with parameter size.	Absolute memorization rates vary wildly (e.g., StarCoder > OLMo1) based on corpus.
Compression	Memorized sequences are highly compressible (≤50% tokens).	Compression depth plateaus at different scales (e.g., 7B vs 13B) depending on the family.
Robustness	LLMs show general denoising capabilities in later layers.	Memorized sequences are more sensitive to noise; the degree of sensitivity varies by family.
Architecture	A small set of attention heads is critical for memorization across domains.	The layer distribution of these important heads is unique to each family (e.g., OLMo vs. Pythia).
Domains	Structural texts are generally easier to memorize.	Larger models shift from memorizing structural text to free-text more aggressively in some families.

5. Significance and Implications

Universal Understanding: The study moves beyond isolated case studies to propose a unified framework for understanding LLM memorization, identifying that while the mechanism (specific heads, two-stage decoding) is universal, the manifestation is dictated by the training recipe.
Privacy & Copyright: The finding that memorized sequences are more sensitive to noise and rely on specific "critical heads" suggests potential avenues for targeted mitigation. Instead of retraining, one could potentially prune or perturb these specific sensitive pathways to reduce memorization without destroying general model capabilities.
Model Design: The identification of "family fingerprints" in layer-wise head distribution implies that memorization structure is an emergent property of the training data and algorithm, not just model size. This informs future model design to either enhance or suppress memorization based on intended use cases.
Data Efficiency: The high compressibility of memorized sequences suggests that models store these patterns in a highly efficient, compressed format, challenging the notion that memorization is merely "overfitting" in a naive sense.

In conclusion, this paper establishes that memorization in LLMs is a complex interplay of universal scaling laws and specific training-induced structural signatures, providing a roadmap for future research into the safety and interpretability of large language models.