💬 NLP

There is No Theoretical Curse of Multilinguality For Embedding Space Structure

This paper theoretically proves that the "curse of multilinguality" is not an inherent limitation of embedding space structure, as the dimensionality required for perfect multilingual alignment grows only logarithmically with the number of languages, suggesting that observed performance degradation stems from real-world data and training conditions rather than theoretical constraints.

Original authors: Niyati Bafna, Neha Verma, Vilém Zouhar, Philipp Koehn, David Yarowsky

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Niyati Bafna, Neha Verma, Vilém Zouhar, Philipp Koehn, David Yarowsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a persistent worry known as the "curse of multilinguality." It describes a frustrating pattern where computer models that try to learn many languages at once often end up performing worse than models that focus on just one. Imagine a student trying to master a dozen subjects simultaneously; the fear is that the sheer volume of information dilutes their attention, leaving them with a shallow understanding of every topic rather than deep expertise in any. In the realm of language technology, this means that as engineers add more languages to a single model, the quality of translation and understanding for each individual language tends to drop. This phenomenon has led many to believe that there is a fundamental limit to how many languages a single machine can handle without becoming a master of none. The core question has been whether this failure is an unavoidable law of nature, a structural flaw in how these digital brains are built, or simply a result of how we feed them data.

A team of researchers from Johns Hopkins University and ETH Zurich has now challenged the idea that this curse is inevitable. They set out to investigate whether the very architecture of these language models—the mathematical space where words and meanings live—has a hard limit on how many languages it can hold. To understand their work, one must first picture the "embedding space." This is not a physical room, but a vast, multi-dimensional map where the computer stores the meaning of words. In this map, words with similar meanings sit close together, while words with different meanings sit far apart. When a model learns multiple languages, it tries to create a single map where the word for "dog" in English sits right next to the word for "dog" in German, French, and Japanese, while still keeping the concept of "dog" distinct from "cat." The prevailing wisdom suggested that as you add more languages to this map, the space would become too crowded, forcing concepts to collide and degrade the model's performance.

The researchers approached this problem by asking a theoretical question: is it mathematically impossible to build a perfect map that holds thousands of languages without the map becoming impossibly large? They defined what a "perfect" multilingual space would look like. In such a space, every language would retain its own high-quality internal structure, meaning the relationships between words within a single language would remain clear and precise. Simultaneously, the space would exhibit perfect alignment, ensuring that the same concept in different languages, like "dog" and "Hund," would be positioned closer to each other than to any unrelated concept. They then used rigorous mathematical proofs to determine the minimum amount of space, or dimensions, required to achieve this perfection as the number of languages grows.

Their findings reveal that the fear of a structural limit is unfounded. The researchers proved that the amount of extra space needed to accommodate more languages does not grow in a straight line or explode exponentially. Instead, it grows very slowly, following a logarithmic curve. To put this in perspective, if you were to double the number of languages, you would not need to double the size of the map; you would only need a tiny, almost negligible increase in capacity. They calculated that even if you wanted to include every language on Earth—estimated at around 7,000—you would theoretically only need about 30 additional dimensions in the model's structure to maintain perfect quality. This suggests that the embedding space itself is not the bottleneck; the structure is inherently capable of scaling to hold the world's languages without collapsing under the weight.

If the structure is not the problem, the researchers turned their attention to why the curse appears to exist in the real world. They conducted a series of controlled experiments, training small computer models on different sets of languages while carefully varying the conditions. They tested scenarios where the total amount of computing power was fixed versus scenarios where the power increased with the number of languages. They also varied how the data was sampled, looking at both uniform distributions where every language got equal attention and realistic distributions that mimicked the uneven availability of data in the real world. The results showed that the "curse" is not a universal law but a consequence of specific training conditions. When the models were trained with sufficient data and compute resources for the target languages, the degradation disappeared. In some favorable configurations, adding more languages actually improved performance, likely because the model learned to transfer knowledge between languages more effectively.

The study concludes that the degradation seen in current multilingual models is not a fundamental flaw in the geometry of language representation. Instead, it is a practical issue arising from how data is distributed and how computing resources are allocated during training. When a model is forced to learn many languages with a fixed, limited budget of data, the performance drops because the model simply does not have enough information to learn each language well. However, the theoretical capacity of the system is far greater than what is currently being utilized. The researchers suggest that the path forward lies not in accepting a limit on language coverage, but in optimizing how data and computing power are shared across languages. By ensuring that every language in a model receives adequate attention and resources, it is possible to build systems that are both massive in scope and high in quality, effectively breaking the curse that has long been thought to be an insurmountable barrier.

Technical Summary: There Is No Theoretical Curse of Multilinguality for Embedding Space Structure

Problem Statement

The "Curse of Multilinguality" (COM) is a well-documented phenomenon in Natural Language Processing where the performance of a fixed-capacity multilingual model degrades as the number of training languages increases. This degradation is often attributed to "capacity dilution," where the model's fixed resources are spread too thinly across too many languages, leading to negative interference. While previous works have explored this empirically through extrinsic task performance (e.g., machine translation, XNLI) and proposed mitigation strategies (e.g., better data sampling, mixture-of-experts), a fundamental theoretical question remains unanswered: Is it inherently impossible to accommodate increasing language coverage in a limited-capacity model while maintaining perfect monolingual quality and cross-lingual alignment? Specifically, does the structure of the embedding space itself impose a theoretical limit that grows prohibitively with the number of languages?

Methodology

The authors approach this problem from a theoretical and intrinsic perspective, focusing on the geometry of embedding spaces rather than extrinsic task performance.

1. Formalizing "Perfect Multilinguality"
The paper defines "perfect multilinguality" through two intrinsic conditions for an embedding space $X_L$ encoding $L$ languages, relative to a high-quality reference concept space $Z$ :

Monolingual Structure: For every language $\ell$ , the monolingual subspace $X_L^\ell$ must exhibit identical similarity rankings to the reference space $Z$ . This ensures high-quality shared semantics within each language.
Cross-lingual Alignment: Translation equivalents of the same concept across different languages must be closer to each other in the embedding space than to any other concept (the "strong" view of alignment).
Non-Degeneracy: The space must maintain a fixed resolution scale, ensuring distinct points remain separable.

2. Theoretical Analysis
The authors investigate the minimum dimensionality required to satisfy these conditions as $L$ increases.

Upper Bound (Constructive Proof): They construct a perfect multilingual space by taking a direct sum of the reference concept space $Z$ (dimension $D_Z$ ) and a "language offset" space $U$ (dimension $D_L$ ). By placing $L$ language offset vectors on a high-dimensional sphere such that they satisfy specific cosine similarity constraints, they prove that $D_L$ grows as $O(\log L)$ .
Lower Bound (Necessity Proof): They demonstrate that any non-degenerate space encoding $L$ languages with fixed angular separation must have a dimensionality that grows at least as $\Omega(\log L)$ . This is derived using spherical cap packing bounds.
Result: The minimum sufficient dimensionality for a perfect multilingual space is $\Theta(\log L)$ .

3. Empirical Validation
To contextualize the theoretical findings, the authors conduct a controlled empirical study using 4-layer Transformer-based BERT encoders trained from scratch.

Configurations: They test eight configurations varying compute constraints (fixed vs. increasing), sampling conditions (uniform vs. realistic), and evaluation sets (all languages vs. a fixed subset).
Metrics: They measure Monolingual Structure via $k$ -nearest-neighbor overlap (MS-NNO) and masked language modeling loss (MS-MLM), and Cross-lingual Alignment via a weak-view alignment metric (CLA-WV).

Key Contributions and Results

Theoretical Contribution:
The paper provides the first theoretical grounding for the curse of multilinguality regarding embedding space structure. The central result is that the dimensionality required to maintain perfect multilinguality grows only logarithmically ( $\Theta(\log L)$ ) with the number of languages.

For a world with $\sim 7,000$ languages, the theoretical "tax" is approximately 30 extra dimensions.
This proves that there is no theoretical curse for embedding space structure; the capacity required does not grow linearly or exponentially, but rather very slowly.

Empirical Findings:
The authors observe an empirical curse of multilinguality in their experiments, but its presence is highly dependent on training and evaluation configurations:

Cursed Configurations: In settings with fixed compute and uniform sampling, performance degrades as languages are added, consistent with the "folk wisdom" of capacity dilution.
Non-Cursed Configurations: In settings where the token count for target languages is maintained (e.g., increasing compute or realistic sampling with a fixed subset of languages), metrics remain stable or even improve.
Conclusion on Empirics: The empirical curse is not an inherent structural limitation but a consequence of real-world constraints (data distribution, compute budgets, and sampling strategies).

Additional Observations:

The authors analyze the "language subspace" in trained models and find that, unlike their ideal constructive solution where language and concept subspaces are orthogonal, empirical models exhibit significant entanglement between language and concept directions.
Even in "non-cursed" empirical settings, the effective dimensionality consumed by language information increases with language count, suggesting current models do not achieve the theoretical ideal of compact, reusable language offsets.

Significance and Claims

The paper claims to offer the first intrinsic and theoretical perspective on the curse of multilinguality. Its primary significance lies in shifting the understanding of the phenomenon:

Theoretical Possibility: It establishes that perfect multilinguality is theoretically achievable at arbitrary language scales without a prohibitive increase in model capacity (dimensionality).
Reframing the Problem: By proving the absence of a theoretical curse, the authors suggest that the observed empirical degradation is a solvable engineering problem related to data, compute, and training dynamics, rather than a fundamental geometric impossibility.
Scientific Clarity: The paper advocates for more precise definitions of "curse" in future work, distinguishing between structural limitations (which do not exist) and empirical limitations (which are sensitive to configuration).

The authors modestly note limitations, acknowledging that their theoretical model assumes a global concept set and that real-world languages may introduce language-specific concepts requiring additional dimensions. Furthermore, their empirical study is limited to small-scale encoder models, leaving the impact of larger architectures and different training regimes for future investigation.

Technical Summary: There Is No Theoretical Curse of Multilinguality for Embedding Space Structure

Problem Statement

Methodology

Key Contributions and Results

Significance and Claims

More like this