💬 NLP

A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs

This paper introduces a typology-aware diagnostic framework using Universal Dependencies and inference-time perturbations to evaluate how multilingual masked language models (mBERT and XLM-R) rely on word order versus inflectional morphology across five languages, revealing that while full scrambling severely degrades performance, lemma substitution significantly impacts accuracy in inflectional languages but fails to mitigate the effects of structural disruption.

Original authors: Anna Feldman, Libby Barak, Jing Peng

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anna Feldman, Libby Barak, Jing Peng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to read and understand sentences in many different languages at once. You want to know: Does this robot understand the meaning of the words, or is it just a master of the order in which they appear?

This paper is like a "stress test" for two very smart robots (called mBERT and XLM-R) to see how they handle languages that play by different rules.

The Two Types of Languages

Think of languages as having two different ways of holding a sentence together:

The "Strict Line-Up" Languages (like English and Chinese): These languages rely heavily on the order of words. If you say "The cat chased the dog," it means one thing. If you say "The dog chased the cat," it means the opposite. The order is the glue.
The "Flexible Puzzle" Languages (like Russian, German, and Spanish): These languages use "word costumes" (inflections/morphology). The words change their endings to show who is doing what to whom. So, even if you scramble the order, the "costumes" on the words tell you the meaning. "The dog" might wear a special hat that says "I am the one being chased," regardless of where it sits in the sentence.

The Experiment: Breaking the Sentence

The researchers wanted to see if the robots were cheating by just memorizing the order, or if they were actually using the "word costumes" to understand meaning. They did this by breaking the sentences in four creative ways:

The "Shuffle Party" (Full Scrambling): They took a sentence and threw all the words into a blender, mixing them up completely.
- Analogy: Imagine taking a deck of cards, shuffling them until they are a random mess, and asking the robot to guess the next card.
- Result: The robots crashed. Their accuracy dropped to almost zero. They couldn't guess the word at all. This proved they rely heavily on the order of words.
The "Keep the Glue" Shuffle (Partial Scrambling): They shuffled the main content words (nouns, verbs) but left the "glue" words (like "the," "is," "and") in their original spots.
- Analogy: You scramble the actors in a play, but you leave the stage directions and the script's punctuation exactly where they were.
- Result: The robots did better than the full shuffle, but still struggled a lot. They missed the mark.
The "Head Swap" (Dependency Swapping): They swapped the main action (the verb) with one of the things it acts on (the noun).
- Analogy: In a sentence like "The chef cooked the soup," they swapped "chef" and "soup" to get "The soup cooked the chef."
- Result: This confused the robots significantly, especially in English.
The "Stripped-Down" Test (+L): They took every word and replaced it with its "dictionary form" (its lemma).
- Analogy: Imagine reading a story where every word is stripped of its tense, gender, or number. Instead of "ran," "runs," or "running," you only see "run." Instead of "she," "he," or "they," you just see "person."
- Result: For languages with rich "costumes" (like German or Russian), this made the robots much worse at guessing. It proved that when the order is messed up, the robots need those word costumes to help them. However, stripping the costumes didn't save them when the order was scrambled.

The Big Surprise

The researchers expected that in languages like Russian or German, the robots would be like expert puzzle solvers: "Even if you scramble the order, I can use the word costumes to figure it out!"

But that's not what happened.

The robots turned out to be obsessed with order.

When the order was broken, the robots failed, even in languages where humans could easily figure it out using the "costumes."
The "word costumes" (morphology) helped a little bit, but they weren't enough to save the robots when the sentence structure was destroyed.
In Chinese (where words don't really have costumes), the robots did okay, but only because the order was the only thing holding it together.

The Takeaway

Think of these multilingual AI models as tourists who only know how to follow a map, not how to read the landscape.

If you give them a map (the word order), they can find their way.
If you burn the map (scramble the order), they get lost, even if the landmarks (the word costumes) are still there and clearly visible.

Why does this matter?
It means these AI models might be biased toward languages like English, where order is king. They might not be as "smart" or "flexible" as we think when dealing with languages that rely on word endings and flexible structures. To make better AI, we need to teach them to look at the "costumes" on the words, not just the line they are standing in.

In short: The robots are great at following the script, but if you change the script's order, they forget the whole play. They haven't truly learned the language; they've just learned the rhythm.

1. Problem Statement

Multilingual transformer models (e.g., mBERT, XLM-R) achieve strong cross-lingual performance, but the specific linguistic cues driving their predictions remain unclear. Languages vary significantly in how they encode grammatical relations:

Fixed-order languages (e.g., English, Chinese) rely on strict word sequencing.
Morphologically rich/free-order languages (e.g., Russian, German, Spanish) rely heavily on inflectional morphology (case, agreement) to determine grammatical roles, allowing for flexible word order.

The core question is whether multilingual models rely primarily on linear word order (risking an English-centric bias) or if they effectively utilize morphological cues to remain robust when word order is disrupted. Existing work is often monolingual (focused on English) or lacks controlled, typologically diverse stress tests.

2. Methodology

The authors propose a typology-aware diagnostic framework that applies inference-time perturbations to multilingual masked language models (MLMs) to isolate the reliance on word order versus morphology.

Models and Languages

Models: mBERT (110M params, WordPiece) and XLM-R (270M params, SentencePiece/BPE).
Languages: Five typologically diverse languages:
- Fixed-order/Isolating: English (EN), Chinese (ZH).
- Fusional/Case-marking: German (DE), Spanish (ES), Russian (RU).
- (Turkish (TR) is included in the appendix but excluded from main analysis due to floor effects).

Perturbation Conditions

Using Universal Dependencies (UD) treebanks, the authors apply controlled perturbations before masking a single content word:

Full Scrambling: Randomly permutes all word tokens. Tests total reliance on sequence.
Partial Scrambling: Permutes only content words (NOUN, VERB, etc.) while keeping function words fixed. Tests if local scaffolding preserves prediction.
Head-Dependent Swaps: Swaps a UD head with one of its dependents (keeping other tokens fixed). Targets predicate-argument anchors.
Lemma Substitution (+L): Replaces every token in the sentence (context and target) with its UD lemma. This acts as an "inflection-stripping" diagnostic to see if models rely on surface forms or abstract lemmas.

Evaluation Protocol

Masking: Applied before perturbation to ensure the same target token is evaluated across all conditions.
Scoring: Word-level exact match (reconstructing the full word from subword predictions).
Metrics:
- Top-1 Accuracy: Exact reconstruction.
- Top-5 Accuracy: Whether the gold word appears in the top 5 candidates (to capture partial knowledge).
- Sensitivity ( $S$ ): Relative drop in accuracy compared to the baseline ($Orig$).
- Interaction ( $I$ ): Measures non-additivity. $I > 0$ suggests overlapping/redundant cues (sub-additive harm); $I < 0$ suggests complementary cues (synergy).

3. Key Contributions

Typologically Grounded Stress Tests: A systematic framework applying structured perturbations across five diverse languages to quantify the trade-off between word order and morphology.
Dependency-Aware Scrambling: Introduction of a "Head" condition that swaps syntactic heads with dependents, offering a more nuanced test than random shuffling.
Lemma Normalization Diagnostic (+L): A novel sentence-level lemma substitution to probe if models can leverage morphological structure when surface forms are normalized.
Comprehensive Benchmarking: Release of code, sampling scripts, and balanced evaluation subsets for mBERT and XLM-R, enabling reproducibility.

4. Key Results

Word Order Dominance

Full Scrambling: Causes a near-total collapse in accuracy across all languages and models (accuracy drops to $\approx 0.01 - 0.02$ ). This indicates that current multilingual MLMs are strongly dependent on linear word order.
Partial & Head Scrambling: Also cause significant accuracy drops, though less severe than full scrambling.
- English/Chinese: Suffer heavily from Head swaps (consistent with fixed SVO order).
- German: Shows slightly more resilience in structured conditions, likely due to function words and case marking, but still suffers significant drops.
- Russian: Despite rich case marking, accuracy remains highly sensitive to word order disruption.

Impact of Lemma Substitution (+L)

No Compensation: Lemma substitution never compensates for lost word order. In fact, it generally lowers accuracy further in morphologically rich languages (DE, ES, RU).
Chinese Exception: In Chinese, +L has negligible effect (near identity) because surface forms closely resemble lemmas. Tiny accuracy gains observed are attributed to tokenization quirks, not genuine morphological leverage.
Conclusion: Models do not switch to a "morphology-only" mode when order is degraded; they become brittle.

Interaction Analysis (Overlap vs. Synergy)

Sub-additive Harm ( $I > 0$ ): In DE, ES, and RU, the combined harm of scrambling + lemma substitution is less than the sum of their individual harms. This suggests that word order and inflectional form provide overlapping, partially redundant cues rather than complementary ones. When one is removed, the other is already degraded or insufficient to maintain performance.
Top-5 Results: Even allowing for 5 candidates, full scrambling pushes the gold word out of the top rankings, confirming that the gold target is not just ranked lower but effectively lost.

5. Significance and Implications

Positional Bias: The study confirms that multilingual transformers are "positional workhorses." They have not learned to robustly exploit morphological cues to the extent that they can overcome the loss of linear order.
Typological Fairness: There is a risk of English-centric bias; models trained on fixed-order languages may fail to generalize to free-order languages when syntactic cues are ambiguous or disrupted.
Future Directions:
- Training Objectives: Suggests the need for objectives that reduce over-reliance on absolute order (e.g., de-scrambling tasks) while preserving syntactic signals.
- Architecture: Proposes morphology-aware prediction heads that explicitly model inflectional structure rather than relying on subword artifacts.
- Evaluation: Calls for constituency-based scrambling and human baselines to better understand the cues humans vs. models use when order is unreliable.

In summary, the paper demonstrates that despite the theoretical capacity of transformers to capture hierarchical syntax, current multilingual MLMs remain heavily reliant on surface-level word order, showing limited robustness when this cue is removed, even in languages where morphology should theoretically suffice.