Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation

1. Problem Statement

Counterfactual examples (minimally edited inputs that flip a model's prediction) are a standard tool for explaining black-box AI models. While Large Language Models (LLMs) have demonstrated impressive capabilities in generating these examples in English, their effectiveness in multilingual contexts remains largely unexplored.

• The Gap: Existing counterfactual generation methods are almost exclusively evaluated on English datasets.
• The Challenge: Multilingual models often exhibit systematic behavioral variations between English and non-English contexts. Relying solely on English counterfactuals fails to capture the full scope of model behaviors across different languages, particularly for low-resource languages.
• Core Question: Can LLMs generate high-quality, valid, and minimal counterfactuals across diverse languages, and how do these compare to English-only or translation-based approaches?

2. Methodology

The authors conducted a comprehensive empirical study involving three LLMs of varying sizes, two datasets, and six languages.

Experimental Setup

• Datasets:
  • XNLI: Cross-lingual Natural Language Inference (Entailment, Contradiction, Neutral).
  • SIB200: Topic Classification (7 topics) across 205 languages.
• Languages: English (en), Arabic (ar), German (de), Spanish (es), Hindi (hi), and Swahili (sw). These were chosen to represent typological diversity and varying resource levels.
• Models: Three open-source instruction-tuned LLMs:
  • Qwen2.5-7B
  • Gemma3-27B
  • Llama3.3-70B
• Generation Strategies:
  1. Directly Generated (DG-CFs): LLMs generate counterfactuals directly in the target language using a three-step Chain-of-Thought (CoT) process: (1) Identify influential words, (2) Find replacements, (3) Substitute.
  2. Translation-Based (TB-CFs): LLMs generate counterfactuals in English first, then translate them into the target language using the same LLM.

Evaluation Metrics

• Label Flip Rate (LFR): The percentage of instances where the counterfactual successfully changes the model's prediction (Validity).
• Textual Similarity (TS): Measured via multilingual SBERT embeddings to ensure minimal edits (Minimality).
• Perplexity (PPL): Measured using mGPT-1.3B to assess fluency.
• Cross-lingual Edit Similarity: Analyzed cosine similarity of edits across languages to determine if LLMs follow consistent strategies.
• Error Analysis: Manual and automatic categorization of failure modes.

Downstream Task: Counterfactual Data Augmentation (CDA)

The study evaluated whether using these generated counterfactuals as training data improves the performance and robustness of a multilingual BERT model. They compared Cross-lingual CDA (training on English counterfactuals only) vs. Multilingual CDA (training on counterfactuals in the target language).

3. Key Contributions & Findings

A. Quality of Multilingual Counterfactuals

• Direct vs. Translation: Translation-based counterfactuals (TB-CFs) generally achieve a higher Label Flip Rate (LFR) than directly generated ones (DG-CFs), suggesting that translating a high-quality English counterfactual is more effective than asking the LLM to generate one from scratch in a low-resource language.
• The Trade-off: However, TB-CFs require substantially more modifications (lower textual similarity) and often suffer from lower fluency due to translation artifacts.
• Resource Disparity: High-resource European languages (German, Spanish) generally perform better than low-resource languages (Arabic, Hindi, Swahili). Notably, English counterfactuals consistently outperform others in LFR, but Hindi often shows superior fluency (lowest perplexity).
• No "Perfect" Language: No single language or model dominates across all metrics (validity, fluency, minimality).

B. Cross-lingual Edit Patterns

• Strategic Similarity: LLMs apply remarkably similar edit patterns across English, German, and Spanish. This suggests that cross-lingual perturbations follow common strategic principles, likely due to structural and lexical similarities among these languages.
• Divergence: Low-resource languages like Swahili and Arabic show significantly different edit patterns and lower cross-lingual consistency, indicating the LLMs struggle to apply the same logic in these contexts.

C. Error Analysis

The study identified four distinct error types prevalent across all languages:

1. Copy-Paste: The LLM returns the original input unchanged. This is the most pervasive error, particularly in SIB200 and for low-resource languages in XNLI.
2. Negation: The LLM simply adds a negation word (e.g., "not") without making deeper semantic changes. This often fails to flip the label or creates ambiguity.
3. Inconsistency: The generated text contains logical contradictions relative to the original context.
4. Language Confusion: The LLM fails to generate in the target language, defaulting to English or mixing languages. This is frequent in low-resource language prompts.

D. Impact on Model Performance (CDA)

• Multilingual > Cross-lingual: Multilingual CDA (training on target-language counterfactuals) generally yields better performance improvements than Cross-lingual CDA, especially for low-resource languages.
• Limitations: Despite the potential, the imperfections of the generated counterfactuals (errors like copy-paste and language confusion) limit the gains. In some cases, adding noisy counterfactuals actually degrades model robustness.
• Error Filtering: When error cases (copy-paste, language confusion) were filtered out, performance gains increased significantly, particularly for low-resource languages like Hindi and Swahili, proving that the noise in the data is the primary bottleneck.

4. Significance and Implications

• Beyond English-Centricity: The paper provides the first large-scale evidence that LLMs are not yet fully capable of generating high-quality counterfactuals in non-English languages without significant degradation in quality or consistency.
• Translation is a Viable (but Imperfect) Strategy: While direct generation in low-resource languages is challenging, translating English counterfactuals offers a viable path to higher validity, provided the translation quality is managed.
• Data Augmentation Risks: Simply augmenting training data with LLM-generated counterfactuals is risky. The study highlights that data quality control (filtering errors) is essential before using these examples for training, especially for low-resource languages.
• Future Directions: The authors suggest that post-training methods (like MAPO) or improved prompting strategies are needed to overcome the inherent biases and limitations of current LLMs in multilingual counterfactual generation.

5. Conclusion

The paper concludes that while LLMs show promise in multilingual counterfactual generation, they currently fall short of English-level performance. The "Parallel Universes" of different languages are not perfectly aligned; edit strategies diverge, and error rates increase in low-resource settings. Future work must focus on improving the fidelity of multilingual generation and developing robust error-correction mechanisms to make counterfactual data augmentation a reliable tool for multilingual AI.