💬 NLP

Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation

This paper demonstrates that multilingual language models underperform their monolingual counterparts in lexical disambiguation due to specific capacity constraints—namely reduced embedding isotropy, diminished attention to disambiguating cues, and increased multi-token segmentation—which collectively explain the observed "multilingual penalty."

Original authors: Sean Trott, Pamela D. Rivière

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sean Trott, Pamela D. Rivière

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Multilingual Penalty"

Imagine you have two students. One student studies only English, while the other studies English, Spanish, French, and German all at the same time. You might think the second student is smarter because they know more languages. However, this paper found that when it comes to understanding tricky, ambiguous words (like the word "lamb," which could mean a baby sheep or a cut of meat), the student who studies all the languages actually performs worse than the student who focuses on just one.

The researchers call this the "Multilingual Penalty." They wanted to figure out why this happens. They suspected it's because the "brain" of the computer model (the Language Model) has a limited amount of space or "capacity." When you try to cram too many languages into that same space, something has to give.

The Experiment: A Test of Context

To test this, the researchers used a specific type of puzzle called Lexical Disambiguation.

The Puzzle: They gave the computer sentences like, "She liked the marinated lamb" versus "She liked the friendly lamb."
The Goal: The computer needs to realize that in the first sentence, "lamb" means meat, and in the second, it means an animal.
The Setup: They compared "Monolingual" models (trained only on English or only on Spanish) against "Multilingual" models (trained on both). They used human judgments as the "correct answer key" to see how well the computers did.

The Result: The multilingual models consistently got the answers wrong more often than the single-language models.

Why Does This Happen? Three Suspects

The researchers investigated three specific ways the multilingual models might be "running out of room." Think of the model's brain as a busy office.

1. The "Crowded Desk" (Representational Constraints)

Imagine a desk where you organize files.

Monolingual Model: Has a huge desk just for English files. They can spread out, and every file has its own clear, distinct spot.
Multilingual Model: Has to fit English, Spanish, and other languages onto that same desk. To make it fit, they have to stack files closer together.
The Problem: When files are stacked too tightly (a lack of "isotropy"), it becomes hard to tell them apart. The computer struggles to distinguish between the "meat" version of a word and the "animal" version because their "files" are too crowded together in the computer's memory.

2. The "Distracted Spotlight" (Attentional Constraints)

Imagine a detective trying to solve a crime. They need to shine a spotlight on the most important clue (the word that tells them if "lamb" is meat or an animal).

The Theory: The multilingual model's "spotlight" (attention mechanism) might be weaker. It's like the detective is trying to solve crimes in two different cities at once; they can't focus as intensely on the clues in one specific sentence as a detective who is only working in one city.
The Finding: In Spanish, the multilingual models definitely shined a dimmer spotlight on the important clues compared to the single-language models.

3. The "Broken Puzzle Pieces" (Vocabulary Constraints)

Imagine you are trying to build a picture using puzzle pieces.

Monolingual Model: Has a box of pieces designed just for English. The word "lamb" fits perfectly into one single piece.
Multilingual Model: Has to fit words from many languages into a box of the same size. To make it work, the word "lamb" might get chopped up into three smaller, weird pieces (like "lam" and "b").
The Problem: When the computer has to glue three small pieces back together to understand the word, it's harder to get the full meaning right. The multilingual models had to break up words into more pieces more often, which confused them.

The Final Verdict: It's Not Just "Being Multilingual"

The researchers ran a final statistical test to see which of these three problems was actually causing the bad performance.

They found that all three problems happened together in the multilingual models. But here is the key discovery:

If you tell a computer, "You are multilingual," it predicts the model will do poorly.
However, if you tell the computer, "This model has crowded files, a weak spotlight, and broken puzzle pieces," it predicts the poor performance even better.

In fact, once you account for those three specific problems, the fact that the model is "multilingual" doesn't matter anymore. The "multilingual penalty" isn't a magic curse; it's just the result of the model trying to do too much with too little space, leading to crowded memories, distracted focus, and broken words.

What This Means (and What It Doesn't)

What it means: Multilingual models do suffer from real limitations when trying to understand the subtle meanings of words. These limitations are measurable and linked to how the computer stores and processes information.
What it doesn't mean: The paper does not say these models are useless, nor does it suggest how to fix them or use them in hospitals or schools. It simply identifies why they struggle with this specific type of word puzzle. The authors also admit their study was limited to English and Spanish and used older model types, so we don't know if this happens exactly the same way in the newest, giant AI models.

Technical Summary: Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation

Problem Statement
While multilingual language models (LMs) are designed to facilitate cross-linguistic generalization, empirical evidence suggests they often underperform their monolingual counterparts as the number of training languages increases. This phenomenon, termed the "multilingual penalty," is hypothesized to stem from insufficient model capacity. The specific challenge addressed in this work is lexical disambiguation, a task requiring precise semantic representations and robust contextualization mechanisms. The authors posit that multilingual LMs may struggle with this task due to three distinct capacity constraints:

Representational: The compression of linguistic knowledge into shared dimensions may reduce embedding isotropy (increasing anisotropy), limiting the model's ability to discriminate between distinct word meanings.
Attentional: The inability of attention heads to specialize for every language may reduce the capacity to attend to disambiguating cues across diverse linguistic contexts.
Vocabulary-related: A fixed vocabulary size covering more languages forces more words to be segmented into multiple sub-word tokens, potentially disrupting coherent representation extraction.

Methodology
The study quantifies the multilingual penalty using controlled datasets of human relatedness judgments for ambiguous words in English (RAW-C) and Spanish (SAW-C). These datasets feature minimal pair contexts where ambiguous words are disambiguated by surrounding cues.

Models: The authors evaluated 24 model instances, including monolingual English, monolingual Spanish, and multilingual variants from the same model families (e.g., BERT/BETO, ALBERT/ALBETO, RoBERTa, DistilBERT).
Performance Metric: Disambiguation performance was measured by regressing human relatedness judgments against the cosine distance between contextualized embeddings of target words across model layers. The resulting $R^2$ served as the index of predictive success.
Capacity Metrics:
- Isotropy: Measured using Centered Isotropy (CI), calculated by averaging cosine distances between normalized embeddings within a sentence.
- Attention: Measured by calculating attention scores between target words and disambiguating cues, aggregating both average and maximum scores per layer.
- Tokenization: Counted the number of sub-word tokens required for target words and disambiguating cues.
Statistical Analysis: Linear mixed models were employed to regress performance ( $R^2$ ) and capacity metrics against factors including model size (log parameter count), layer depth, language, and multilingual status.

Key Results

Existence of the Penalty: Multilingual models consistently exhibited reduced disambiguation performance compared to equivalent monolingual models, even when controlling for parameter count and layer depth ( $\beta = -0.16, p < .001$ ).
Capacity Constraints Observed:
- Isotropy: Multilingual models showed significantly reduced mean Centered Isotropy, a deficit that worsened in deeper layers.
- Attention: In the Spanish dataset, multilingual models demonstrated reduced maximum attention to disambiguating cues in later layers compared to monolingual counterparts. (No significant difference was found in the English dataset).
- Tokenization: Multilingual models consistently required a higher number of tokens to represent both target words and disambiguating cues.
Accounting for Variance: When modeling disambiguation performance, the combination of isotropy, attention, and tokenization metrics provided a substantially better fit than a baseline model. Crucially, when these three factors were included in the full model, the "Multilingual status" variable became statistically unnecessary, indicating that these capacity constraints fully account for the variance previously attributed to multilingualism.
- Increased isotropy predicted higher performance ( $R^2$ ).
- Increased multi-token segmentation predicted lower performance.
- Note: Contrary to initial hypotheses, increased attention to disambiguating cues was associated with decreased performance in the final model, a finding the authors attribute to potential issues with operationalizing attention or distributed attention patterns.

Significance and Claims
The paper claims to provide a proof-of-concept that the multilingual penalty in lexical disambiguation is not merely a black-box phenomenon but correlates with measurable, interpretable capacity constraints. The primary contribution is the statistical demonstration that reduced isotropy, reduced attentional focus on cues, and increased token fragmentation collectively explain the performance gap between multilingual and monolingual LMs.

The authors maintain a modest stance, acknowledging that their findings are correlational and do not establish causality. They note limitations in scope (restricted to English and Spanish, using non-state-of-the-art model families) and the coarse definition of "multilingual" status. Consequently, the paper does not propose specific architectural fixes or applications but rather identifies specific dimensions (isotropy, attention, tokenization) that future research should investigate to understand and potentially mitigate the multilingual penalty.