💬 NLP

Vocabulary shapes cross-lingual variation of word-order learnability in language models

This study demonstrates that the structure of word and subword vocabulary, rather than broad typological distinctions like free versus fixed word order, is the primary factor determining the learnability of word-order variations in transformer language models.

Original authors: Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann, Lisa Beinborn

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann, Lisa Beinborn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to speak different human languages. You want to know: Is it harder for the robot to learn some languages than others? And specifically, why can some languages (like Czech) scramble their words around like a deck of cards, while others (like English) demand you put the words in a strict, unchangeable order?

This paper is like a scientific experiment where the researchers built a "language gym" to test these robots. Here is the story of what they found, explained simply.

1. The Experiment: The "Scramble Machine"

The researchers didn't just pick random languages. They took 10 real European languages (from English and French to Finnish and Czech) and created 100 different "versions" of each one.

Think of it like taking a sentence and running it through a scramble machine.

Version A: The words are in perfect, normal order.
Version B: The words are slightly jumbled (like a sentence where you swapped the first two words).
Version C: The words are completely mixed up (random chaos).
Version D: The sentence is read backward.

They then trained a small AI robot (a language model) on each of these scrambled versions to see how confused the robot got. In AI terms, "confusion" is called surprisal. If the robot is surprised, it means the sentence is hard to learn. If it's not surprised, the sentence is easy.

2. The Big Surprise: Chaos is Hard, But Backwards is Easy

The first thing they found was intuitive: The more you scramble the words, the harder it is for the robot to learn.

If you take a sentence and mix up the words randomly, the robot's brain lights up with confusion. It's like trying to read a book where the letters are all mixed up; it takes much more effort to figure out the meaning.

However, there was a twist.
The researchers thought that reading a sentence backward (e.g., "cat the paints robot the") would be the hardest thing of all. But it wasn't! The robot handled backward sentences almost as well as slightly jumbled ones.

Analogy: Imagine you are trying to solve a puzzle. If you take the pieces and throw them in a box (random chaos), it's a nightmare. But if you just flip the whole picture upside down (reversal), the robot can still figure out the shapes. It turns out, the robot doesn't care as much about the direction of the sentence as it does about the randomness of the pieces.

3. The Old Theory Was Wrong

For a long time, linguists thought languages fell into two neat boxes:

Fixed Order Languages: Like English (Subject-Verb-Object).
Free Order Languages: Like Czech (where you can swap words freely because the words have special "tags" or endings that tell you who is doing what).

The researchers expected that the "Free Order" languages would be naturally tougher for the robot to scramble because they rely on those word tags. But they were wrong.

When they tested the robots, the "Free Order" languages and the "Fixed Order" languages performed almost exactly the same when scrambled. The old "two-box" theory didn't explain the differences.

4. The Real Hero: The Vocabulary "Backpack"

So, if the word order isn't the main culprit, what is? The answer lies in the vocabulary—specifically, how the words are built and how many of them there are.

Imagine every language has a backpack of words.

English has a backpack with many simple, short words.
Finnish or Hungarian has a backpack with fewer, but very long and complex words (like a single word that means "I will not have been able to do it").

The researchers found that the structure of this backpack is what determines how hard a language is to learn when scrambled.

The Zipf Effect: In every language, a few words are used all the time (like "the," "and," "is"), and most words are used very rarely.
The Discovery: Languages where the "rare words" are very rare (a steep drop-off in usage) were harder for the robot to learn when scrambled.
The Metaphor: Think of a language with complex words as a Lego set with thousands of tiny, unique pieces. If you shake the box (scramble the order), it's very hard to rebuild the castle because you need to find those specific, rare pieces. A language with simple, repetitive words is like a Lego set with big, standard bricks. Even if you shake the box, it's easier to rebuild because the pieces are common and interchangeable.

5. The Conclusion

The paper concludes that it's not the "rules" of word order that make a language hard to learn for AI; it's the "ingredients" (the vocabulary).

Simple takeaway: If you want to know how hard a language is for a computer to learn, don't ask "Can I move the words around?" Instead, ask "How complex are the words themselves, and how many rare words are there?"
The "Backpack" matters more than the "Order": A language with a heavy, complex backpack of words is harder for a robot to juggle than a language with a light, simple backpack, regardless of whether the words are allowed to dance around or stand in a line.

In short: Vocabulary structure is the secret sauce. It's the hidden force that decides whether a language is a breeze for a robot to learn or a brain-bending puzzle.

1. Problem Statement

The paper addresses a fundamental question in computational linguistics: Why are some languages computationally harder to learn than others? Specifically, it investigates the relationship between word-order flexibility (the degree to which words in a sentence can be reordered without changing meaning) and the learnability of a language by Transformer-based language models (LMs).

While natural languages exhibit a spectrum from "fixed" word order (e.g., English) to "free" word order (e.g., Czech, Finnish), previous research has struggled to isolate these factors due to:

Correlated Typological Features: Word order is often entangled with morphological complexity in natural languages.
Methodological Limitations: Prior perturbation studies often shuffled at the subword level, which inadvertently breaks morphological integrity (splitting words into linguistically implausible fragments) and alters global text entropy, confounding the results.
Discrete vs. Continuous: Typological classifications (free vs. fixed) are often treated as binary categories, whereas linguistic reality is a gradient.

The authors aim to disentangle the effects of word-order regularity, vocabulary structure, and morphological complexity on model surprisal (a proxy for learnability).

2. Methodology

A. Synthetic Language Generation

The authors create a continuous spectrum of synthetic language variants for 10 European languages (5 fixed-order: English, French, Portuguese, Swedish, Danish; 5 free-order: Czech, Latvian, Hungarian, Estonian, Finnish) based on the Europarl corpus.

Deterministic Word-Level Shuffling: Unlike previous subword shuffling, they shuffle whole words to preserve morphological integrity.
Mallows Permutation Model: They use the Mallows $\phi$ $ϕ$ model to control the degree of disorder via a single continuous parameter, $\theta$ :
- $\theta \to \infty$ : Original word order.
- $\theta > 0$ : Local shuffling (minor deviations).
- $\theta = 0$ : Completely irregular (uniform distribution over permutations).
- $\theta < 0$ : Aversion to original order, leading to sentence reversal as $\theta \to -\infty$ .
Control: For each sentence length $n$ , a single permutation is sampled and applied deterministically to all sentences of that length. This ensures the global entropy of the text remains constant, isolating the effect of word-order structure.

B. Experimental Setup

Models: Lightweight autoregressive Transformer models (PICOLM framework, ~50M parameters) trained from scratch on each synthetic variant.
Tokenization: ByteLevel-BPE with a vocabulary size of 16,000 (varied in specific experiments).
Evaluation Metric: Surprisal ( $S$ ), defined as the average negative log-probability of the next token.
- $\Delta S(\theta) = S(\theta) - S_{orig}$ : The change in surprisal relative to the original language serves as the measure of learnability degradation. Lower $\Delta S$ implies higher robustness to perturbation.

C. Analysis Techniques

Multivariate Partial Least Squares (PLS) Regression: Used to model the relationship between vocabulary statistics (predictors) and surprisal curves (responses) across the 10 languages. This handles high collinearity among predictors (e.g., sentence length and morphological complexity).
Leave-One-Language-Out Cross-Validation: To test the generalizability of vocabulary-based predictors.

3. Key Contributions

Methodological Innovation: Introduction of a deterministic, word-level shuffling framework using the Mallows model. This preserves morphological structure and global entropy, solving the confounds present in prior subword-shuffling studies.
Refutation of Binary Typology: Demonstrates that the binary distinction between "free" and "fixed" word-order languages is insufficient to explain cross-lingual differences in learnability.
Vocabulary as the Primary Driver: Identifies vocabulary structure (specifically Zipfian coverage metrics, sentence length, and morphological complexity proxies) as the strongest predictor of a language's robustness to word-order perturbations.
Asymmetry in Reversal: Shows that while irregular word order significantly increases surprisal, sentence reversal has a surprisingly weak effect on learnability, suggesting models are not strongly sensitive to the specific typological correlations violated by reversal.

4. Key Results

A. Word-Order Irregularity Increases Surprisal

As word order becomes more irregular (approaching $\theta = 0$ ), model surprisal ( $\Delta S$ ) increases significantly across all languages.
Sentence Reversal ( $\theta < 0$ ): Surprisingly, reversing sentences yields a surprisal change similar to positive-order perturbations. The models are largely insensitive to the specific violation of typological correlations caused by reversal, indicating a lack of strong inductive bias against reversed syntax compared to random shuffling.

B. The Failure of Binary Classification

When plotting $\Delta S$ at $\theta=0$ against baseline surprisal, "free" and "fixed" languages overlap significantly.
Statistical tests (Wilcoxon–Mann–Whitney) show no significant difference in robustness between the two groups when treated as binary categories.

C. Vocabulary Structure Explains Variance

Coverage Metrics: Languages with "free" word order tend to have lower coverage by high-frequency words/subwords (slower Zipfian decay) compared to "fixed" languages.
PLS Regression: A model using vocabulary statistics (coverage, sentence length, fertility, word types) explains 97% of the variance ( $R^2 = 0.97$ $R^{2} = 0.97$ ) in surprisal curves across languages.
- Component 1 (Vocabulary/Coverage): Explains surprisal for original and reversed orders.
- Component 2 (Morphological Complexity): Specifically explains robustness against irregular word orders ( $\theta \approx 0$ ). Languages with complex morphology (high fertility, longer words) are more sensitive to shuffling.
Vocabulary Size Sensitivity: Increasing vocabulary size reduces the surprisal gap for fixed-order languages but has a different effect on free-order languages, further highlighting the role of tokenization and subword distribution.

5. Significance and Implications

Redefining Learnability: The paper argues that computational learnability is not determined by coarse typological labels (e.g., "SVO" vs. "SOV") but by the statistical properties of the vocabulary and how tokenization interacts with morphological complexity.
Architectural Insights: The results suggest that autoregressive models rely heavily on local context and vocabulary predictability. The insensitivity to reversal implies that models may not be learning deep syntactic dependencies but rather statistical regularities in token sequences.
Future Directions: The authors suggest that future work should investigate how architectural choices (masking, positional encoding) and human cognitive data (eye-tracking) align with these computational findings. The study provides a robust framework for generating synthetic languages to probe specific linguistic hypotheses without the noise of natural language variation.

In conclusion, the paper establishes that vocabulary structure is a key driver of computational word-order learnability, offering a more nuanced and predictive framework than traditional typological classifications.