💬 NLP

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

The paper introduces MultiBLiMP 1.0, a fully automated, massively multilingual benchmark comprising over 128,000 minimal pairs across 101 languages that evaluates large language models' subject-verb agreement capabilities and exposes significant shortcomings in modeling low-resource languages.

Original authors: Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to speak 101 different languages. You might think, "Great! It can chat with anyone!" But there's a hidden problem: just because a robot can tell a funny story in French doesn't mean it actually understands the grammar rules of French. It might be guessing based on patterns it saw in its training data, rather than truly "knowing" the rules.

This paper introduces MultiBLiMP 1.0, a massive new test designed to see if these AI robots actually know the grammar rules, or if they are just faking it.

Here is a breakdown of what they did, using simple analogies:

1. The "Grammar Police" Test (Minimal Pairs)

Imagine you have two sentences that are almost identical, like twins:

Sentence A: "The cat sleeps." (Correct)
Sentence B: "The cat sleep." (Incorrect)

The only difference is one tiny word. A human knows instantly that Sentence B is wrong. But does the AI know?

MultiBLiMP is a giant collection of over 128,000 of these "twin sentences" (called minimal pairs) covering 101 languages. The test is simple: The AI looks at the pair and has to guess which one is the "good" sentence. If it picks the wrong one, it means it doesn't understand the rule (in this case, that a singular cat needs a singular verb).

2. The Factory That Built the Test

Creating these tests manually for 101 languages would take humans years. Instead, the authors built a fully automated factory.

They used two giant digital libraries of language data (Universal Dependencies and UniMorph) as their raw materials.
They programmed a pipeline (a step-by-step assembly line) to find sentences, identify the grammar rules, and then automatically "break" one of the sentences to create the wrong version.
Think of it like a robot chef that takes a perfect cake, swaps out one ingredient to make it taste bad, and then asks the AI, "Which one is the real cake?"

3. The Results: Big vs. Small, and Pre-Training vs. Post-Training

The researchers tested 42 different AI models (from small ones to massive ones) on this grammar test. Here is what they found:

Size Matters (The "Library" Analogy): Generally, bigger models (with more "brain power" or parameters) did better. It's like a student who has read more books is more likely to know the grammar rules.
The "Language Diet" Matters: Models performed best on languages that were very common in their training data (like English or Spanish). If a language was rare in the data (like a low-resource language), the models struggled, even if they were huge. It's like a chef who only knows how to cook Italian food; if you ask them to cook a rare dish from a specific village in the Andes, they will likely mess it up.
The "Fine-Tuning" Trap: This was a surprising finding. The researchers compared the "raw" AI (after it learned from the internet) with the "polished" AI (after humans taught it how to be helpful and follow instructions).
- The Raw AI was actually better at grammar.
- The Polished AI got worse at grammar.
- Analogy: Imagine a brilliant student who knows all the grammar rules. Then, they go to a "customer service school" to learn how to be polite and helpful. When they come back, they are great at being nice, but they've forgotten some of the strict grammar rules they used to know. The "polishing" process accidentally made them worse at the grammar test.

4. Why This Matters

The paper argues that we need to stop just asking AIs to "write a poem" or "translate a document" to see if they are smart. Those tasks are too messy; the AI can get the right answer for the wrong reasons.

MultiBLiMP is like a pure math test for language. It strips away the need for world knowledge or creativity and asks a simple question: "Do you know the rules?"

The Bottom Line

The paper concludes that while our AI models are getting better at talking to us, they are still shaky on the fundamental rules of grammar, especially for languages that aren't very common on the internet. Furthermore, the process of making these models "helpful" (chat mode) might actually be hurting their ability to understand strict grammar rules.

The authors hope this tool will help researchers build better models that don't just sound fluent, but actually understand the structure of language.

1. Problem Statement

Large Language Models (LLMs) are increasingly trained on massive, multilingual corpora. However, current evaluation methods for these models suffer from two main limitations regarding formal linguistic competence (knowledge of grammatical rules and statistical regularities):

Coarse-grained Evaluation: Existing multilingual benchmarks often rely on generative tasks (translation, summarization) or perplexity. These conflate formal competence with functional competence (world knowledge) and fail to isolate specific syntactic failures.
English-Centric Minimal Pairs: While "minimal pair" benchmarks (comparing grammatical vs. ungrammatical sentence variants) exist for English (e.g., BLiMP), they are scarce for other languages. Existing non-English benchmarks are often monolingual, manually curated, or limited to high-resource languages, preventing a scalable assessment of how LLMs handle syntax across diverse linguistic typologies.

The authors aim to bridge this gap by creating a massively multilingual, automated benchmark to evaluate the formal syntactic abilities of LLMs, specifically focusing on subject-verb agreement.

2. Methodology

The core contribution is MultiBLiMP 1.0, a pipeline for the automatic generation of linguistic minimal pairs across 101 languages. The system leverages two large-scale linguistic resources: Universal Dependencies (UD) and UniMorph.

The Pipeline (4 Stages)

Candidate Extraction:
- The system parses UD treebanks to identify subject-verb and subject-participle relations.
- It applies filters to ensure valid syntactic structures (e.g., removing sentences with reparandum dependencies, foreign words, or clausal subjects that complicate minimal pair formation).
- It targets specific morphological features: Number, Person, and Gender.
Agreement Validation (Data-Driven Filter):
- To ensure the generated "ungrammatical" sentences are actually violations in the target language, the system calculates conditional probabilities of feature co-occurrence (e.g., $P(\text{Verb}=PL | \text{Subject}=PL)$ ).
- It performs statistical significance testing (binomial test) to confirm that a language exhibits strict agreement for a specific feature. If a language allows optional agreement (e.g., Turkish plural marking), that specific condition is discarded to avoid false negatives.
Inflection:
- Once a valid candidate is found, the system uses UniMorph to generate the inflected form of the verb that violates the agreement rule (e.g., changing a singular verb to plural).
- It handles lemma disambiguation and transliteration to ensure the generated form is morphologically correct but syntactically mismatched with the subject.
Dataset Balancing:
- To prevent bias toward specific lexical items or sentence structures, the dataset is downsampled and balanced.
- The system ensures an even distribution of features such as subject-verb distance, attractor nouns (intervening material), and morphological feature values.

Scope

Languages: 101 languages (covering 14 language families).
Data Volume: 128,321 minimal pairs (derived from an initial pool of 1.4 million).
Phenomena: Subject-Verb Agreement and Subject-Participle Agreement across Number, Person, and Gender.

3. Key Contributions

Massive Multilingual Scale: The first benchmark of its kind covering 101 languages, significantly expanding beyond the English-centric BLiMP and other monolingual efforts.
Fully Automated Pipeline: A novel, language-agnostic pipeline that generates minimal pairs without human intervention, relying on UD and UniMorph. This allows for rapid expansion to other linguistic phenomena.
Rigorous Validation: A data-driven statistical approach to verify that generated minimal pairs are genuine grammatical violations in the target language, addressing the "optional agreement" problem common in cross-lingual evaluation.
Comprehensive Evaluation: Evaluation of 42 LLMs (including Llama 3, Gemma 3, Qwen 3, OLMo 2, Aya, and EuroLLM) across both base (pre-trained) and chat (post-trained) versions.

4. Experimental Results

The authors evaluated 42 models on the 128k+ minimal pairs. Key findings include:

Model Size Correlation: Linguistic competence is strongly driven by model size. Larger models (e.g., Llama-70B, Gemma-27B) consistently outperform smaller ones.
Impact of Post-Training (Instruction Tuning): Counter-intuitively, post-training (chat versions) often degrades formal syntactic competence compared to the base pre-trained models. For example, Llama-70B-chat performed worse than Llama-70B-base on many languages.
Data Frequency Bias: Performance is heavily correlated with the frequency of the language in the training data (Common Crawl). Low-resource languages show significantly lower accuracy.
Monolingual vs. Multilingual: Goldfish (a suite of monolingual models trained on equal amounts of data) significantly outperformed large multilingual LLMs on 14 out of 101 languages, particularly low-resource ones. This suggests that multilingual LLMs struggle when a language represents a tiny fraction of the training mix.
Tokenization Issues: The "subword delta" (difference in token count between correct and incorrect forms) negatively impacts performance, suggesting tokenizer inefficiencies contribute to errors.
Specific Phenomena: Models perform best on Person agreement (often involving closed-class pronouns) and worst on Gender and Number agreement (requiring open-class feature acquisition).

5. Significance and Future Implications

Diagnostic Tool: MultiBLiMP 1.0 provides a granular view of where LLMs fail syntactically, moving beyond "black box" generative metrics.
Training Recommendations: The results suggest that for low-resource languages, targeted regional monolingual initiatives may be more effective than integrating these languages into massive multilingual mixes, as the latter dilutes formal competence.
Interpretability: The benchmark offers a unified framework for studying how LLMs learn and represent grammatical rules across typologically diverse languages, aiding in the development of better interpretability tools.
Resource Validation: The work demonstrates the continued critical value of traditional linguistic resources (UD, UniMorph) in the era of LLMs, proving they are essential for rigorous, rule-based evaluation.

In conclusion, MultiBLiMP 1.0 reveals that while modern LLMs possess impressive multilingual capabilities, their formal syntactic competence is fragile, heavily dependent on data volume, and often degraded by instruction tuning, highlighting a need for more balanced and targeted training strategies for low-resource languages.