💬 NLP

Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures

This paper demonstrates that Large Language Models generalize better across languages when their latent representations accurately capture the hierarchical similarity structures of language families, such as the Indo-European tree, with this structural alignment directly correlating to improved performance on multilingual benchmarks like XNLI.

Original authors: Supantho Rakshit, Adele Goldberg, Henry Conklin

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Supantho Rakshit, Adele Goldberg, Henry Conklin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world. You don't just give it a dictionary; you let it read millions of books, websites, and stories. Over time, the robot starts to build a mental map of how things relate to one another. In the world of computer science, this is called a "Large Language Model" (LLM). Think of these models as super-smart digital brains that have read almost everything on the internet. But here's the tricky part: most of what they read is in English. When you ask them to speak a language they haven't seen much of, like Swahili or Icelandic, they often stumble.

Scientists have long wondered why this happens. A key idea from psychology suggests that our brains (and smart computers) learn by noticing similarities. If you know what a "dog" is, you can guess what a "wolf" is because they look and act alike. This is called "generalization." The better your mental map groups similar things together, the better you are at guessing new things. But does this same rule apply to entire languages? Do these digital brains secretly organize languages the way a family tree organizes relatives? That is the big question this paper sets out to answer.

The Great Language Family Reunion

Picture a giant, invisible party where every language in the Indo-European family (a huge group including English, Spanish, Hindi, and Russian) is invited. In the real world, we know these languages are related like cousins, siblings, and distant relatives. Some, like Spanish and Italian, are like twins who grew up in the same house. Others, like English and Hindi, are distant cousins who haven't seen each other in thousands of years.

The researchers in this paper wanted to see if Large Language Models (LLMs) secretly know this family tree. They didn't ask the models, "Hey, are these languages related?" Instead, they looked at the models' "thoughts" (their internal math) to see how they grouped languages together. It's like watching a guest at the party and seeing who they stand next to. If the model puts Spanish and Italian close together, but keeps them far away from Hindi, it's showing it understands the family connections without ever being explicitly taught them.

The Discovery: The Models Have a Secret Map

The team tested 12 different AI models, ranging from older ones to newer, more powerful ones. They fed the models thousands of sentences in 38 different languages. Then, they used a special mathematical trick to map out how the models "felt" about each language.

The result was surprising and delightful: The models had accidentally built a map that looked almost exactly like the real family tree of languages.

Just like a genealogist drawing a family tree, the models naturally grouped languages into the same sub-families: Germanic languages (like English and German) huddled together; Romance languages (like French and Spanish) formed their own circle; and Slavic languages (like Russian and Polish) stuck to their own group. The models didn't need a teacher to tell them, "These are Romance languages." They figured it out on their own just by reading the data.

Why Some Models Are Better at the Party

But here is the twist: not all models are equally good at this. The researchers found a direct link between how well a model organized its language map and how well it actually performed on a difficult test called XNLI (a test where the model has to guess if one sentence proves, contradicts, or has nothing to do with another sentence in a different language).

Think of it like this: If a student has a messy, jumbled notebook where they mix up all their history facts, they will struggle to answer a test question. But if their notebook is perfectly organized by topic, they can find the answer instantly. The paper found that models which organized their "language notebook" correctly (keeping similar languages close together) were much better at solving the test problems.

Specifically, the models that were trained on data from many different languages (multilingual models) did a much better job of drawing this family tree than models trained mostly on English. It's as if the multilingual models had a bigger, clearer map, while the English-only models were trying to navigate with a blurry, incomplete sketch.

The "Script" Trap and Other Surprises

One might guess that the models just grouped languages by how they look on paper (their alphabet or script). For example, maybe they thought Hindi and Urdu were similar just because they both use the Devanagari script? The paper says no. While the way words are written (orthography) does have a small effect, it's not the main reason. The models were smart enough to see that Hindi and Urdu are linguistic cousins, even though they use different scripts, and that Persian (which uses an Arabic script) is actually related to North Indian languages, not Arabic ones. The models were looking at the structure of the language, not just the font.

However, the models weren't perfect. They sometimes got a bit confused with "orphan" languages like Greek, Welsh, and Albanian. In some models, Greek and Welsh were paired up strangely, even though they aren't closely related. This suggests that while the models are great at seeing the big picture, they sometimes trip over the unique, isolated branches of the family tree.

When Did They Learn This?

The researchers also watched the models as they were being trained, like watching a baby learn to walk. They found that the models started organizing languages into these family groups very early in their training—after just 100 steps of learning. This suggests that figuring out "who is related to whom" is one of the first things a smart language model learns, before it even gets into the nitty-gritty details of specific words.

The Bottom Line

This paper suggests that the secret to a language model's success isn't just memorizing more words; it's about building a good mental map of how languages relate to each other. When a model understands that Spanish and Italian are close cousins, and that English and German are siblings, it becomes much better at translating ideas from one to the other.

It turns out that these digital brains are surprisingly good at history. They didn't need a textbook to learn the Indo-European family tree; they just needed to read the world, and the patterns emerged on their own. And the better they get at drawing that map, the better they get at speaking every language on it.

Technical Summary: Similarity All The Way Up

Problem Statement
While Large Language Models (LLMs) demonstrate impressive capabilities across diverse tasks, their ability to generalize—particularly multilingually to low-resource languages outside of English—remains poorly understood. Current models often struggle with languages poorly attested in training data. The paper posits that this limitation stems from the representational structures learned by LLMs. Drawing on cognitive science, the authors argue that successful generalization depends on the underlying geometry of similarity spaces (Shepard, 1957; Tenenbaum & Griffiths, 2001). Specifically, the paper investigates whether LLMs implicitly learn the hierarchical similarity structures of natural languages (e.g., the Indo-European family tree) and whether the fidelity of these learned structures correlates with cross-lingual transfer performance.

Methodology
The study analyzes the representational geometry of 12 LLMs across five families (BERT, RoBERTa, mT5, XLM, and Pythia), including both monolingual and multilingual variants.

Data and Reference Tree: The analysis focuses on 38 living Indo-European languages across five subfamilies (Romance, Germanic, Indo-Aryan, Slavic, Baltic) and independent languages. A "Reference Tree" is constructed using the Bayesian Maximum Clade Credibility (MCC) tree from Heggarty et al. (2023), which estimates hierarchical similarity based on lexical cognate judgments. This serves as the ground truth for language relationships.
Representation Extraction: For each model, the authors estimate the conditional density of language representations in the embedding space. Using a soft-density estimator (Conklin, 2025), they discretize continuous hidden states into a codebook via temperature-scaled softmax. This generates a language-specific conditional distribution $P(\hat{Z}|\text{language})$ for each layer.
Distance Calculation: Pairwise Jensen-Shannon (JS) divergences are computed between language distributions across all layers to create a distance matrix. UPGMA clustering is applied to this matrix to induce a dendrogram representing the model's internal similarity structure.
Alignment Metrics: Three metrics are used to compare the model-induced dendrogram against the Reference Tree:
- Cophenetic Correlation (CC): Measures the preservation of pairwise distance relationships (global fit).
- Path Kernel: Measures shared root-to-leaf paths, capturing global hierarchical nesting (e.g., subfamily membership).
- Subtree Kernel: Measures exact local groupings and topology (strictest test).
Performance Correlation: The alignment scores are correlated with model performance on the XNLI (Cross-lingual Natural Language Inference) benchmark, which measures zero-shot transfer capabilities.

Key Results

Implicit Recovery of Phylogeny: LLMs implicitly recover the hierarchical structure of the Indo-European language family. Models group languages belonging to the same subfamily (e.g., Germanic, Romance) closely in representation space without explicit supervision.
Correlation with Performance: There is a significant positive correlation between the degree to which a model's representation aligns with the Reference Tree and its XNLI accuracy. Specifically, the Cophenetic Correlation ( $\rho=0.81$ ) and Path Kernel ( $\rho=0.79$ ) show strong positive relationships with performance. The Subtree Kernel does not correlate significantly ( $\rho=0.38$ ), suggesting that capturing broad, higher-order subfamily structures is more critical for generalization than matching exact nearest-neighbor relationships.
Impact of Training Data: Multilingual models (mBERT, XLM-Roberta) align significantly better with the Reference Tree than their monolingual counterparts (BERT, RoBERTa), despite sharing identical architectures.
Emergence Timing: Analysis of Pythia model checkpoints reveals that language-level similarity structures begin to emerge early in pre-training (by gradient step 100), aligning with theories of progressive differentiation where broad distinctions are learned before subtle properties.
Orthography vs. Linguistics: While tokenization is sensitive to script (e.g., Cyrillic vs. Latin), linguistic relatedness remains the dominant signal. For instance, Hindi and Urdu (different scripts) are grouped correctly, while Persian (Arabic script) does not migrate to the Semitic cluster despite script overlap.

Significance and Claims
The paper claims to extend the theoretical framework of similarity-driven generalization from the level of words and concepts to the level of entire languages.

Theoretical Extension: It demonstrates that the geometric organization of representations, a core principle in cognitive science and connectionist models, scales to multilingual LLMs.
Predictive Power: The study establishes that the fidelity of a model's internal language similarity structure is a predictor of its cross-lingual generalization ability. Models that represent similar languages similarly generalize better.
Mechanism of Transfer: The results suggest that multilingual performance relies on the model's ability to align related semantic concepts across languages by recovering the hierarchical similarity structure of the language family, rather than merely sharing vocabulary.

The authors maintain a modest scope, noting that while the results are robust for Indo-European languages, systematic errors (such as unexpected pairings of Greek and Welsh) warrant further investigation. They do not propose new training algorithms but rather offer a diagnostic lens for understanding how representational geometry drives multilingual generalization.