💬 NLP

MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference

This paper introduces MERGE, a test that uses Masked Language Models to automatically generate minimal expression replacements of existing Natural Language Inference datasets, revealing that current state-of-the-art reasoning models struggle to generalize robustly to these non-adversarial variants.

Original authors: Mădălina Zgreabăn, Tejaswini Deoskar, Lasha Abzianidze

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mădălina Zgreabăn, Tejaswini Deoskar, Lasha Abzianidze

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student to take a driving test. You show them a red car, and they correctly say, "That's a car." You show them a blue car, and they say, "That's a car." They pass the test with flying colors.

But then, you ask a tricky question: "If I change the word 'car' to 'truck' in your answer, does it still make sense?" Or, "If I change the color from 'red' to 'blue' in the description, does the logic still hold?"

This is exactly what the paper MERGE (Minimal Expression-Replacement GEneralization) is about. It's a new way to test if AI models are actually "smart" at reasoning, or if they are just memorizing patterns like a student who only knows the answers to the specific practice questions they studied.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Memorizer" vs. The "Thinker"

Current AI models are very good at Natural Language Inference (NLI). This is a task where you have two sentences:

Premise: "A girl carries a small girl."
Hypothesis: "There is a small girl."
Task: Does the first sentence prove the second one? (Yes).

The models get almost 100% on standard tests. But the authors suspect these models are like students who memorized the exact words on the test. They aren't really understanding the logic; they are just spotting that "girl" appears in both sentences.

2. The Solution: The "Minimal Swap" Test (MERE)

The authors created a new test called MERGE. Instead of giving the AI a totally new, confusing problem, they take a problem the AI already knows and make tiny, minimal changes.

Think of it like a game of "Spot the Difference" with a twist:

Original: "A girl carries a small girl."
The Swap: The AI is asked to handle: "A boy carries a small boy." or "A girl carries a happy girl."

The rules of the game (called MERE generation) are strict:

Keep the logic: If the original sentence meant "Yes," the new one must also mean "Yes."
Keep the structure: Don't change the grammar or the length of the sentence.
Keep the overlap: The words that were shared in the original must still be shared in the new one.

They use a tool called a "Masked Language Model" (think of it as a very smart word-suggestion engine, like the one in your phone's keyboard) to suggest these swaps automatically.

3. The New Scoring System: The "Consistency Check"

In a normal test, if you get 90% right, you pass. But in the MERGE test, the scoring is stricter.

Imagine you have one original question (the Seed) and 20 slightly different versions of it (the Variants).

Old Way: If the AI gets 18 out of 20 right, it gets a high score.
MERGE Way: The AI must get all (or almost all) of the 20 variants right to count the original question as "solved."

If the AI gets the original right but fails on the "boy" version, it means the AI was just memorizing the word "girl," not understanding the concept of "someone carrying someone."

4. The Results: The AI Crumbles

The authors tested this on many different AI models, from smaller ones to massive "Large Language Models" (LLMs) like the ones you might chat with today.

The findings were surprising:

The "Memorizers" Failed: When the AI had to handle these tiny changes, its performance dropped significantly. It could handle the original questions, but as soon as the words changed slightly, it got confused.
The "Consistency" Gap: Even the best models could only consistently handle about 50% of the variants. If the test required them to be consistent on 60% or more of the changes, their scores plummeted.
The Hardest Words: The study found that changing verbs (actions) was the hardest for the AI, followed by nouns (things), and then adjectives (descriptions). It seems the AI struggles most when the action changes.
No "Magic" Source: They wondered if the AI performed better if the word suggestions came from the same type of AI that was taking the test. They found no difference. It didn't matter where the new words came from; the AI still struggled to generalize.

5. The Conclusion

The paper concludes that current AI models are brittle. They are excellent at solving the specific problems they were trained on, but they lack true "generalization." They haven't learned the deep rules of logic; they've just learned to recognize specific patterns.

When you tweak the pattern just a little bit (like swapping "girl" for "boy"), the AI's confidence and accuracy break down. The authors call this the MERGE test, and it shows that we still have a long way to go before AI can truly "reason" like a human, rather than just "pattern match" like a parrot.

In short: The AI is great at reciting the script, but if you ask it to improvise with a single word change, it freezes.

Technical Summary: MERGE – Minimal Expression-Replacement GEneralization

Problem Statement
Current Natural Language Inference (NLI) models often achieve high performance on standard test sets by exploiting heuristics learned from training data, such as word overlap or hypothesis-only artifacts, rather than genuine reasoning capabilities. While Out-of-Distribution (OOD) datasets have been used to expose these weaknesses, existing approaches often rely on adversarial examples that deliberately break common heuristics or use non-syntax-preserving modifications that introduce implausibility. Furthermore, manually constructing high-quality reasoning datasets is costly and difficult to scale, while automatic generation methods have historically suffered from bias or the creation of ungrammatical variants. There is a need for a robust, automated method to evaluate model generalization against non-adversarial, minimal changes that preserve the underlying logical reasoning and syntactic structure of the original problems.

Methodology: MERE and MERGE
The authors propose MERGE (Minimal Expression-Replacement GEneralization), a framework for evaluating model robustness, built upon MERE (Minimal Expression REplacement), an automated generation technique.

MERE Generation Process:
- Seed Selection: The process starts with an original NLI "seed" problem consisting of a premise ( $p$ ) and a hypothesis ( $h$ ) with a label $l$ .
- Target Identification: The system identifies open-class words (nouns, verbs, adjectives, adverbs) shared between $p$ and $h$ .
- Replacement via MLMs: Masked Language Models (MLMs) are used to suggest contextually probable replacements for these shared words.
- Constraints: To ensure validity, replacements ( $r$ $r$ ) must satisfy three conditions:
  1. $r$ has a higher probability than the original word ( $o$ ) in the masked context under the MLM.
  2. $r$ does not already occur in the sentence.
  3. $r$ belongs to the same part-of-speech (word class) as $o$ .
- Cross-Validation: Replacements are validated across both $p$ and $h$ to ensure they are suggested by the same model for multiple occurrences within a sentence and are consistent across the pair.
- Quality Filtering: A multi-stage filtering process removes subwords, punctuation, and low-probability candidates. Manual validation confirms fluency and the preservation of the original inference relation.
- Dataset Construction: For each seed, multiple variants are generated by replacing shared words. The final dataset (Var) is constructed by randomly sampling variants to avoid over-representation of highly productive seeds.
MERGE Evaluation Metric:
- Unlike standard sample-based accuracy, MERGE introduces Seed-Variant (SV) accuracy.
- A seed problem is considered correctly classified only if a minimum Correctness Threshold (CT) of its generated variants are also correctly classified.
- The Matching Correctness (MC) threshold is defined as the CT at which a model's SV score most closely matches its standard seed accuracy. A lower MC indicates poorer generalization.

Key Contributions

Automated Methodology: A fully automated pipeline (MERE) to create "friendly" generalization test sets that preserve reasoning, syntax, and word overlap while introducing minimal lexical diversity.
Empirical Findings on Generalization: Evidence demonstrating that both fine-tuned NLI models and Large Language Models (LLMs) generalize poorly to minimal variants. Models struggle to maintain consistency when variants differ only slightly in form.
Linguistic Analysis: Identification that verbs are generally the most difficult word class to generalize to, followed by nouns and adjectives.
Source Independence: Findings indicating that the specific source MLM used to generate the variants does not significantly bias the performance of the evaluated NLI models.

Results

Generalization Gap: While models perform well on individual variants (standard accuracy), their performance drops significantly when evaluated on the consistency of variants per seed (SV accuracy). Most models maintain their original performance only when the correctness threshold is set to approximately 50% or lower. Beyond this threshold, performance declines sharply, particularly for LLMs.
Dataset Comparison: Models showed slightly better generalization on MNLI variants compared to SNLI variants.
Word Class Sensitivity:
- Verbs: Consistently the hardest to generalize to, especially at higher correctness thresholds.
- Nouns: Generally harder than adjectives but easier than verbs in MNLI; harder than verbs in SNLI at high thresholds.
- Adjectives: Generally the easiest class to generalize to.
MLM Source Impact: The origin of the replacement words (e.g., using BERT to generate variants for a BERT-based NLI model) did not lead to systematically higher or lower scores, suggesting the generation method does not inherently bias the evaluation toward specific model architectures.
Seed Difficulty: Models performed significantly worse on seed problems they initially misclassified, with only ~80% of their variants being correct, compared to ~97% for correctly classified seeds.

Significance and Claims
The paper claims that MERGE provides a rigorous, non-adversarial test for generalization that reveals weaknesses in current state-of-the-art models that standard metrics obscure. By enforcing consistency across minimal, label-preserving variants, the study demonstrates that models rely heavily on specific lexical cues rather than robust reasoning. The authors position MERGE as a tool to reveal "new types of weaknesses" in reasoning models without the need for costly manual dataset construction or the introduction of adversarial noise. While currently applied to NLI, the authors note the methodology is applicable to other reasoning tasks.

Limitations
The authors acknowledge several limitations:

The method does not replace words tokenized into subwords to avoid probability aggregation artifacts.
When a word appears multiple times, only one instance is masked at a time, which could introduce bias (though such cases are rare in the seed datasets).
Morphological derivations are not replaced to avoid errors in nominal compounds, which can occasionally lead to low-fluency variants.
The analysis did not include larger LLMs beyond those tested.