💬 NLP

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

This study demonstrates that controlled lexical and syntactic perturbations significantly degrade the performance and destabilize the rankings of large language models across major benchmarks, revealing a reliance on surface-level patterns over robust linguistic competence and highlighting the critical need for robustness testing in standard evaluations.

Original authors: Bogdan Kostić, Conor Fallon, Julian Risch, Alexander Löser

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bogdan Kostić, Conor Fallon, Julian Risch, Alexander Löser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a cooking competition. You have 23 different chefs (the AI models) and you want to see who makes the best dish. Usually, you give them the exact same recipe and ingredients, taste the result, and rank them from 1st to 23rd place.

This paper is about what happens when you secretly tweak the recipe just a tiny bit—without changing the actual taste or the final dish—and see if the judges (the AI) get confused.

Here is the story of their experiment, broken down simply:

1. The Setup: The "Same Dish, Different Menu"

The researchers took three famous "recipe books" (benchmarks) used to test AI:

MMLU: A general knowledge quiz (like a trivia night).
SQuAD: A reading comprehension test (finding answers in a story).
AMEGA: A medical guideline test (checking if a doctor follows the rules).

They asked 23 different AI chefs to solve these tests. But then, they created two special "tricks" to see if the AI was actually smart or just memorizing the menu:

Trick #1: The Synonym Swap (Lexical Perturbation).
Imagine the recipe says, "Add a large pinch of salt." The researchers changed it to, "Add a big pinch of salt." The meaning is identical, but the words are different.
Trick #2: The Sentence Shuffle (Syntactic Perturbation).
Imagine the recipe says, "The chef chopped the onions." They changed it to, "The onions were chopped by the chef." The action is the same, but the sentence structure is flipped.

2. The Big Surprise: The AI is a "Word-Matcher," Not a "Thinker"

When the researchers ran the tests, they found something shocking:

The AI hates new words: When they swapped words (like "large" to "big"), the AI's performance crashed. It was like a student who memorized the answer key but failed the test because the teacher wrote "big" instead of "large." The AI seemed to rely heavily on specific words it had seen before, rather than understanding the concept.
The AI is okay with sentence flips: When they rearranged the sentence structure, the AI was much more stable. It could handle the shuffle better than the word swap.

The Analogy: Think of the AI like a parrot. If you teach a parrot to say, "The sky is blue," it might get confused if you ask, "Is the sky blue?" or "Blue is the sky." It's reacting to the specific sounds it heard, not the idea of the sky. This paper shows that current AIs are acting more like parrots than deep thinkers.

3. The Leaderboard is a "House of Cards"

In the AI world, there are public leaderboards (like a high-score list) that tell us which AI is the "best."

The researchers found that these leaderboards are brittle.

On the general knowledge test (MMLU), the rankings stayed mostly the same.
But on the reading and medical tests, the rankings flipped wildly.
- A model that was ranked #1 might drop to #12 just because someone changed a few words.
- A model that was ranked #14 might jump to #9 just because the sentence structure changed.

The Metaphor: Imagine a race where the runners are ranked by who crosses the finish line first. But then, the referee changes the color of the finish line tape from red to blue. Suddenly, the runner who was second wins, and the winner loses. The paper says our current AI leaderboards are like that race—they are unstable and can't be trusted if the "tape color" (the wording) changes slightly.

4. Bigger Isn't Always Better

There is a common belief in tech that "bigger models are smarter and tougher." The researchers tested this by comparing tiny AIs to massive ones.

The Result: Size doesn't guarantee safety.
- On the trivia test, the biggest models actually got the worst scores when the words were changed. They were so specialized that they broke easily.
- On the reading test, the bigger models were actually more robust.

The Analogy: It's like a bodybuilder. A bodybuilder might be incredibly strong at lifting heavy weights (big model on a specific task), but if you ask them to do a delicate dance move (a different task), they might be clumsier than a lightweight gymnast. Being "big" doesn't make you good at everything.

The Takeaway: What Should We Do?

The paper concludes that we are currently overestimating how "smart" these AIs are. They are very good at spotting patterns in the words they've seen before, but they aren't truly understanding the meaning.

The Solution:
Before we trust an AI with important jobs (like diagnosing diseases or writing legal contracts), we shouldn't just ask, "What is your score?" We should ask, "How do you handle it if I say the same thing in a different way?"

We need to start testing AI for robustness (toughness) just as much as we test them for accuracy (correctness). If an AI breaks when you change a synonym, it's not ready for the real world.

1. Problem Statement

The rapid proliferation of Large Language Models (LLMs) has made standardized benchmarks (e.g., MMLU, SQuAD) the primary instrument for model comparison and selection. However, the reliability of these benchmarks is increasingly questioned due to prompt sensitivity. Prior research indicates that superficial changes to input prompts—such as adding irrelevant context, reordering premises, or altering answer choice labels—can significantly degrade performance or shift model rankings.

A critical gap remains in understanding the source of this sensitivity: Is performance variance driven primarily by lexical shifts (changes in word choice) or syntactic alterations (changes in grammatical structure)? Current evaluation methods often fail to distinguish between these two, potentially overestimating the true generalization capabilities and abstract linguistic competence of LLMs.

2. Methodology

The authors conducted a systematic evaluation of 23 state-of-the-art LLMs (ranging from ~0.3B to ~235B parameters, including open-weight and proprietary models) across three diverse benchmarks:

MMLU: Multiple-choice question answering (Humanities, STEM, Social Sciences).
SQuAD: Extractive question answering (Wikipedia articles).
AMEGA: Clinical guideline adherence (Free-form medical QA).

The study employed two linguistically principled, meaning-preserving perturbation pipelines to generate truth-conditionally equivalent variations of the datasets:

Lexical Perturbation Pipeline:
- Uses a quantized Llama-3.3-70B-Instruct to perform guided synonym substitution.
- Ensures contextual appropriateness and preserves domain-specific terminology.
- For SQuAD, a strict constraint ensures the contiguous answer string remains unaltered to isolate lexical effects from answer extraction.
- Outputs are constrained via JSON schemas to ensure consistent formatting.
Syntactic Perturbation Pipeline:
- Uses dependency parsing (via spaCy) to identify matrix clause constituents (subjects, objects, complements).
- Applies rule-based transformations (e.g., active-to-passive, extraposition, wh-movement, dative alternation) only when specific grammatical conditions are met.
- A second LLM stage executes the transformation using operation-specific templates to ensure grammaticality while preserving propositional content.
- Edits are restricted to the matrix clause to avoid altering embedded logic.

Evaluation Protocol:

Models were tested in a zero-shot setting with temperature set to 0 for determinism.
Metrics: Accuracy (MMLU), Exact Match/F1/Semantic Answer Similarity (SQuAD), and Guideline Adherence Score (AMEGA).
Statistical Analysis: McNemar's test for intra-model significance, Wilcoxon signed-rank test for distribution shifts, and Kendall's $\tau$ for ranking stability.

3. Key Contributions

The paper establishes three primary findings that challenge current assumptions in LLM evaluation:

Lexical Dominance over Syntactic Structure: LLMs are significantly more sensitive to lexical variations (synonym substitution) than to syntactic rephrasing. This suggests models rely heavily on surface-level word patterns rather than abstract grammatical competence.
Brittleness of Leaderboards: Meaning-preserving perturbations are sufficient to destabilize model rankings, particularly on complex tasks. Static leaderboards are shown to be unreliable for model selection as minor linguistic changes can alter the "best" model.
Non-Monotonic Scaling of Robustness: The assumption that "bigger is better" regarding robustness is false. The relationship between model size and stability is task-dependent:
- On MMLU, larger models showed greater fragility (positive correlation between size and performance drop).
- On SQuAD and AMEGA, larger models showed increased robustness (negative correlation).

4. Results

Performance Degradation:
- Lexical perturbations caused substantial, statistically significant performance drops across nearly all models and tasks.
  - MMLU: Average accuracy drop of 7.72 percentage points (pp).
  - SQuAD: Average F1 drop of 3.38 pp.
- Syntactic perturbations had heterogeneous effects, with smaller average drops (e.g., 1.64 pp on MMLU) and occasionally even improving results for specific models.
- Significance: 22 out of 23 models showed highly significant ( $p < 0.001$ ) performance drops on MMLU due to lexical changes.
Ranking Instability:
- MMLU: Rankings remained relatively stable under both perturbation types (Kendall's $\tau \approx 0.98$ for lexical, $0.99$ for syntactic).
- SQuAD & AMEGA: Rankings were highly volatile. For example, under lexical perturbation on SQuAD, GPT-4.1-Nano dropped from rank 8 to 12, while GPT-OSS-120b improved from 14 to 9.
- Kendall's $\tau$ for SQuAD and AMEGA dropped to 0.87–0.93, failing to meet the threshold for "strict equivalence" ( $\tau > 0.9$ ) in many cases.
Model Size vs. Robustness:
- MMLU: Strong positive correlation ( $r=0.84$ ) between model size and performance drop. Larger models were less robust.
- SQuAD/AMEGA: Negative correlations ( $r=-0.47$ and $-0.67$ respectively). Larger models were more robust on these tasks.

5. Significance and Implications

Re-evaluating Generalization: The findings suggest that current LLMs may be "overfitting" to the specific lexical patterns of benchmark test sets rather than developing true abstract reasoning or linguistic competence.
Benchmark Design: Static leaderboards are insufficient for high-stakes model selection. The paper advocates for robustness testing (evaluating performance under meaning-preserving perturbations) to become a standard component of LLM evaluation.
Training Methodologies: The results highlight the need for new training strategies, such as data augmentation with perturbed inputs or semantic consistency losses, to penalize models for differing outputs on synonymous inputs.
Reproducibility: The authors released the perturbed datasets and implementation code to facilitate further research into linguistic generalization and model stability.

In conclusion, the paper argues that the field must move beyond static scores toward a "veridical" evaluation paradigm that prioritizes stability and robustness against superficial linguistic variations.