FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation

1. Problem Statement

Current Natural Language Processing (NLP) evaluation paradigms primarily rely on test sets that share the same distribution as training data. While this measures in-domain performance, it fails to reveal model brittleness (fragility) when faced with minor, linguistically valid variations. Existing robustness benchmarks often suffer from:

• Task-specificity: Tests are designed for specific tasks (e.g., sentiment analysis) and do not generalize well.
• Adversarial focus: Many tests rely on "corruption" (e.g., random character swaps, typos) rather than natural, fluent linguistic variations.
• Limited diversity: Manual creation of contrastive pairs is labor-intensive and often lacks the breadth to cover all linguistic phenomena.
• Misleading conclusions: Models may appear robust on standard benchmarks but fail on minimal contrastive pairs or out-of-distribution inputs.

The paper argues that there is a need for a task-agnostic, linguistically-driven framework that systematically generates minimal modifications to test data to uncover these hidden weaknesses.

2. Methodology: The FLUKE Framework

FLUKE (Framework for Linguistically-Driven and Task-Agnostic Robustness Evaluation) is a systematic approach to generating minimal modifications of existing test data based on linguistic principles.

A. Linguistic Test Suite

FLUKE covers a hierarchy of linguistic levels, from low-level orthography to high-level dialect and style:

1. Orthography: Spelling changes (addition, omission, swapping), capitalization (lower, upper, sPoNgEcAsE), and punctuation.
2. Morphology: Derivation (changing root words to derived forms) and compounding.
3. Syntax: Voice changes (active/passive), grammatical role swapping (subject/object), and conjunction modifications.
4. Semantics: Synonym replacement, hypernym/hyponym shifts, nonce words (non-existent words), idioms, and various forms of negation (verbal, absolute, approximate, lexical, double).
5. Discourse: Manipulation of discourse markers and appraisal (sentiment-bearing words).
6. Language Varieties: Style changes (casual vs. formal) and dialect shifts (specifically Singaporean Creole English/Singlish).
7. Bias Tests: Temporal bias (old-fashioned words), geographical bias (underrepresented regions/names), and length bias.

B. Generation and Validation Pipeline

• Automated Generation: Large Language Models (LLMs), specifically GPT-4o, are prompted with templates to generate these modifications for any given input text.
• Human Validation: To ensure quality and validity, a two-stage human annotation process is employed (using Prolific):
  • Stage 1: Annotators verify that the modification is fluent, minimal, and correctly targets the intended linguistic feature.
  • Stage 2: Annotators re-label the modified samples to determine if the ground-truth label changes (e.g., negation flipping sentiment). Samples where the label becomes ambiguous are rejected.
• Metric: Robustness is measured via an "Unrobustness" score ($U$).
  $$U = \frac{1}{N} \sum_{i=1}^{N} |m_i - o_i| \times 100$$
  Where $o_i$ and $m_i$ are correctness indicators for the original and modified samples. A change in prediction (even if it corrects a previous error) counts as unrobustness, indicating the model relies on surface patterns rather than true understanding.

C. Experimental Setup

• Tasks: Six diverse NLP tasks were evaluated:
  • Classification: Named Entity Recognition (NER), Coreference Resolution, Dialogue Contradiction Detection, Sentiment Analysis.
  • Generation: Instruction-Following Evaluation (IFEval), Mathematical Reasoning (GSM8K).
• Models:
  • Pre-trained Language Models (PLMs): BERT, GPT-2, T5 (fine-tuned).
  • Base LLMs: GPT-4o, Llama 3.1 (405B), Claude 3.5.
  • Reasoning LLMs: GPT-5, DeepSeek R1.

3. Key Contributions

1. FLUKE Framework: A novel, task-agnostic system that automates the generation of linguistically diverse, minimal modifications using LLMs, validated by humans.
2. Comprehensive Evaluation: The first large-scale application of such a framework across both classification and generation tasks, covering a wide spectrum of linguistic phenomena.
3. Open Resources: Release of prompt templates, evaluation scripts, validated datasets, and a dashboard for comparing model performance on original vs. modified instances.
4. Empirical Insights: A set of counter-intuitive findings regarding model brittleness, scaling laws, and the relationship between generation capability and robustness.

4. Key Results and Findings

A. Task-Dependent Brittleness

The impact of linguistic variations is highly task-dependent.

• NER: Highly sensitive to Geographical Bias and Capitalization (models rely on capitalization cues for entities). Surprisingly, robust to negation.
• Coreference: Highly sensitive to Syntax (Voice, Grammatical roles) and Negation.
• IFEval (Instruction Following): Highly sensitive to Negation, Style, and even minor changes like adding an appraisal word or changing a single word's case.
• GSM (Math): Highly sensitive to Negation (models fail to update logic) and Geographical Bias (failing to adapt to new names/currencies).

B. LLMs vs. PLMs and Reasoning Models

• LLMs vs. PLMs: While LLMs generally show better overall robustness than fine-tuned PLMs, they are still significantly brittle.
• Reasoning Models: Surprisingly, reasoning-enhanced models (e.g., DeepSeek R1, GPT-5) are not consistently more robust than base LLMs. In some tasks (like NER and Dialogue Contradiction), reasoning models performed worse or similarly to base models.
• Scaling: Increasing model size (e.g., Llama 8B $\to$ 70B $\to$ 405B) improves robustness primarily for surface-level modifications (orthography, simple syntax). However, scaling offers diminishing returns for complex semantic changes, particularly negation, which remains a major failure point even for the largest models.

C. Natural vs. Adversarial Modifications

• Natural Modifications are More Damaging: Linguistically valid, fluent modifications (e.g., changing style, dialect, or negation) generally reveal more unrobustness than traditional adversarial corruptions (e.g., random punctuation or spelling errors).
• Exception: For NER, orthographic changes are critical; for IFEval, capitalization changes are surprisingly disruptive.

D. Generation vs. Robustness Decoupling

• No Correlation: A model's ability to generate a specific linguistic feature (e.g., writing in Singlish or using negation) does not correlate with its robustness to that feature in downstream tasks.
• Circularity Check: The authors verified that GPT-4o could easily generate modifications (high retain rate) that it subsequently failed to handle robustly when tested, proving that generation capability does not imply understanding.

5. Significance and Conclusion

The paper highlights that current NLP models, including state-of-the-art LLMs and reasoning models, have not achieved true linguistic generalization. They often rely on statistical correlations and surface patterns rather than deep semantic understanding.

Significance:

• Diagnostic Tool: FLUKE provides a necessary diagnostic tool to identify specific weaknesses in models before deployment, moving beyond simple accuracy metrics.
• Model Cards: The authors suggest integrating these robustness metrics into "Model Cards" to provide a more transparent view of model capabilities and limitations.
• Future Research: The framework challenges the community to move beyond standard benchmarks and develop training strategies that specifically address semantic robustness (especially negation and reasoning) rather than just scaling parameters.

In summary, FLUKE demonstrates that robustness is not a monolithic property but a complex, task-specific attribute that current models struggle to master, particularly when faced with natural, fluent linguistic variations.