💬 NLP

Language models struggle with compartmentalization

This paper demonstrates that large language models often fail to unify distinct presentations of the same latent concept (such as different languages or programming paradigms) into shared internal representations, leading to redundant learning, reduced sample efficiency, and a phase transition in intervention effectiveness based on the number of concept presentations.

Original authors: Thomas Vincent Howe, David Wingate

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Vincent Howe, David Wingate

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Parallel Universe" Problem

Imagine you have a student who is incredibly smart but has a very strange habit: they treat the exact same fact as if it were two completely different, unrelated secrets depending on how it's written.

If you tell them, "The sky is blue" in English, they write it down in their "English notebook."
If you tell them, "The sky is blue" in Swahili, they write it down in a separate "Swahili notebook."
They never realize these two notebooks are talking about the same thing. They don't cross-reference them. They don't say, "Oh, I already know this!"

This paper calls this behavior Compartmentalization.

The authors argue that Large Language Models (LLMs) often do this. Instead of building one giant, efficient brain that understands the concept of "the sky is blue," they build separate, redundant brains for every language, programming code, or style of writing they encounter. This wastes their brainpower (capacity) and makes them slower to learn (sample inefficiency).

The Experiment: The "Magic Vocabulary" Trick

To prove this happens, the researchers created a controlled experiment. They didn't just use different languages; they used a "magic trick" to force the model to see the same data as totally different.

The Analogy:
Imagine a library where every book is written in the same language, but the librarian has a rule:

If a book is on the Left Shelf, the word "Apple" is written as "Apple."
If a book is on the Right Shelf, the word "Apple" is written as "Banana" (even though it means the same thing).

The researchers created a model where the "Left Shelf" and "Right Shelf" had completely different dictionaries. They fed the model the exact same stories, just with the words swapped according to the shelf.

The Result:
The model failed to connect the dots. It learned the story for the Left Shelf, and it learned the story for the Right Shelf, but it treated them as two totally separate tasks.

The Cost: Because it was learning the same thing twice, it needed more data to get good at it (Sample Inefficiency).
The Bottleneck: Because it was storing two copies of the same knowledge, it ran out of "brain space" faster (Capacity Cost).

The "Translation" Trap

The researchers tried a common fix: they gave the model "translation pairs." They showed it, "Here is 'Apple' on the Left Shelf, and here is 'Banana' on the Right Shelf."

The Surprise:
The model learned the translation rule almost instantly. It could perfectly translate "Apple" to "Banana." However, this didn't help it stop compartmentalizing. It still kept the two concepts in separate mental boxes. It was like a student who can translate a sentence perfectly but still doesn't understand that the underlying meaning is the same.

They found that this only started to work when they gave the model massive amounts of translation data, or when they used a specific training technique called "weight decay" (which acts like a gentle nudge to simplify the model's thinking).

The "Phase Transition"

The paper discovered something weird called a Phase Transition.
Think of it like a light switch.

If you have 2 compartments (Left/Right shelves), the model stays stubborn and doesn't share knowledge.
If you have 8 compartments, suddenly, with enough translation data, the model "snaps" and starts sharing knowledge across all of them.

It seems the model only realizes it should unify its brain when the problem gets too complicated to keep separate.

Real-World Examples

The researchers tested this on real-world scenarios to see if it happens outside their "magic library":

Multilingual Learning: They trained small models on English and Chinese. They found that for small models, the English and Chinese knowledge didn't really mix. The model was essentially "compartmentalized," treating them as separate tasks rather than one unified understanding.
Biographies vs. Q&A: They gave the model facts about people in two formats:
- Format A: A biography paragraph ("John is a doctor who lives in New York").
- Format B: A Q&A pair ("Who is John? A doctor in New York").
- The Disaster: When the model tried to learn both formats at once, they fought each other. The model got confused and performed worse than if it had only learned one format. It was like trying to hold two heavy weights in one hand; they canceled each other out.

The Good News: It's Fixable

The most important finding is that this isn't a fundamental flaw in how AI works; it's just a bad habit the model falls into.

The researchers showed that if you manually copy the knowledge from one compartment to another at the start (or after training), the model performs perfectly. This proves that the model is capable of sharing knowledge, but the standard training process (SGD) just doesn't naturally figure out how to do it efficiently.

Summary

The Problem: AI models often treat the same fact presented in different ways (different languages, different formats) as completely unrelated tasks.
The Consequence: This wastes their memory and makes them slower to learn.
The "Translation" Failure: Simply showing the model how to translate between formats isn't enough to fix this; they need a lot more data or specific training nudges to realize the concepts are the same.
The Takeaway: AI models are currently "compartmentalized." They have separate rooms in their brain for every way a fact can be said, rather than one big room where all the facts live together. The paper suggests we need better training methods to help them merge these rooms.

Technical Summary: Language Models Struggle with Compartmentalization

Problem Statement

Large Language Models (LLMs) are trained on data containing the same latent concepts presented in multiple distinct ways (e.g., facts in English and Swahili, functions in Python and Haskell, or propositions in formal and natural language). While it is desirable for models to recognize these as related and share statistical strength, frontier models often fail to do so.

The paper defines compartmentalization as a phenomenon where a model treats $c$ distinct presentations of a unified latent data generation process as partially or completely separate tasks. In the worst case, the model learns parallel internal representations for each presentation without amortizing information between them. This leads to two primary inefficiencies:

Sample Inefficiency: Each presentation benefits only from its own share of the data. If a presentation accounts for a fraction $\epsilon$ of the data, the model may require $1/\epsilon$ times more samples to learn the structure compared to a unified presentation.
Capacity Competition: Redundant representations consume finite model capacity. A compartmentalized model may saturate sooner than a cohesive model, plateauing at a higher validation loss even with increased data.

Methodology

To isolate and study compartmentalization, the authors propose a "worst-case" experimental construction that removes natural structural cues (like subword overlap or syntactic regularity) that typically encourage representation sharing.

The Tokenization Trick

The authors create $c$ distinct presentations of statistically identical data by modifying the tokenizer:

The base vocabulary size $V$ is increased by a factor of $c$ to create a joint vocabulary of $cV$ tokens.
To encode data in presentation $j$ , token IDs are offset by $jV$.
This ensures that the same underlying text is represented by completely disjoint sets of tokens across compartments, forcing the model to learn $c$ independent mappings unless it discovers a shared structure.

Experimental Setup

Models: GPT-style decoders trained on $\sim$ 131 billion tokens of FineWeb.
Scales: Experiments cover six model sizes ranging from 1.1M to $\sim$ 1B parameters.
Training: Models are trained to convergence (overtrained relative to Chinchilla-optimal) to observe capacity limits.
Baselines: Comparisons are made between a single-compartment model ( $c=1$ ) and compartmentalized models ( $c \in \{2, \dots, 8\}$ ).

Key Contributions and Results

1. Quantifying the Costs of Compartmentalization

The authors demonstrate that compartmentalization imposes significant costs that scale with the number of compartments ( $c$ ):

Sample Efficiency Slowdown: As $c$ increases, the number of training steps required to reach a specific validation loss increases linearly. For example, at $c=8$ , the model requires significantly more steps to reach the same loss as the $c=1$ baseline.
Capacity Costs: Final validation loss plateaus at higher values as $c$ increases, indicating that the model cannot represent the data as efficiently when forced to maintain disjoint representations.

2. Existence of Unified Solutions

The paper proves that the inability to share strength is not a fundamental limitation of the architecture or the data, but rather a failure of the optimization process (SGD) to find specific parameter settings:

Initialization Duplication: If the model is initialized by copying the embeddings of the base vocabulary across all $c$ compartments (rather than random initialization), it achieves sample efficiency matching the $c=1$ baseline.
Post-hoc Parameter Duplication: A trained $c=1$ model can be converted into a $c$ -compartment model by duplicating its learned embeddings. After a brief initial loss spike, the model resumes training on the original performance plateau without capacity overhead.
Implication: Unified solutions exist and are reachable, but standard training from random initialization fails to discover them.

3. Limitations of Standard Interventions

The authors test common strategies for encouraging representation sharing and find they are often insufficient or exhibit phase transitions:

Paired "Translation" Data: Introducing data where the same text appears in two compartments (paired with a translation task) does not reduce compartmentalization for small $c$ $c$ . The model learns the translation task (predicting tokens in one compartment given the other) to near-zero loss, yet the compartments continue to compete for capacity.
- Phase Transition: A reduction in compartmentalization (breaking the loss plateau) only occurs after a $c$ -dependent threshold is crossed (e.g., at $c=8$ with high translation ratios).
- Weight Decay: Aggressive weight decay accelerates this phase transition, shifting the threshold to lower values of $c$ (e.g., from $c=8$ down to $c=6$ ).
Contrastive Alignment (InfoNCE): An auxiliary objective to align middle-layer representations of the same text across compartments shows similar behavior. It provides no measurable benefit for $c=2$ , modest benefits for $c=4$ , and significant recovery of the "compartmentalization tax" for $c=6$ and $c=8$ .

4. Case Studies on Natural Data

The authors apply their framework to real-world scenarios to validate the concept:

Multilingual Training: Training on English and Chinese Wikipedia data shows that small models (<1B parameters) exhibit high compartmentalization. The performance of a shared-tokenizer multilingual model closely tracks a "compartmentalized" baseline (using disjoint tokenizers), suggesting low cross-presentation information sharing in early training stages.
Synthetic Biographies vs. Q&A: A synthetic task presenting facts as either biographical paragraphs or Q&A pairs reveals destructive capacity competition. When trained jointly with a shared tokenizer, the model performs poorly on both formats compared to training on a single format. However, when the formats are compartmentalized (disjoint tokenizers), performance on the biographical format improves significantly, indicating that the joint training failure was due to interference in shared parameters rather than a lack of capacity.

Significance and Claims

The paper argues that the language modeling objective may only inconsistently unify representations across distinct presentations of the same concept. The key takeaways are:

Compartmentalization is a Real Phenomenon: LLMs frequently fail to share statistical strength between distinct presentations (languages, formats, code), leading to redundant learning and wasted capacity.
Optimization Failure: The failure to unify representations is not due to architectural incapacity; unified solutions exist and are reachable via specific initializations, yet standard SGD training does not find them.
Phase Transitions in Learning: Interventions like translation data or contrastive losses do not linearly improve performance. Instead, they exhibit phase transitions where effectiveness only emerges after a certain threshold of data ratio or model complexity ( $c$ ) is reached.
Implications for Efficiency: The findings suggest that current training paradigms may be inherently sample-inefficient and capacity-inefficient when dealing with multi-faceted data, as models may be "learning from each other" far less than assumed.

The authors remain modest, noting that their findings are based on small-scale models and that compartmentalization might diminish at larger scales, though existing evidence from frontier models suggests the phenomenon persists. They propose their tokenizer-based construction as a baseline for measuring representation sharing in future research.