Analyzing Language Bias Between French and English in Conventional Multilingual Sentiment Analysis Models

1. Problem Statement

Despite advancements in Natural Language Processing (NLP), significant biases persist, particularly regarding the underrepresentation and unequal performance across different languages. While English is often the "high-resource" language with superior model performance, other languages like French may suffer from disparities in sentiment analysis accuracy and fairness.

The specific problem addressed is the language bias in conventional machine learning models (specifically Support Vector Machines and Naive Bayes) when applied to multilingual sentiment analysis. The study investigates whether, even when datasets are balanced (50-50 split between English and French), inherent biases favor one language over the other, and whether standard fairness metrics can detect these nuances.

2. Methodology

Dataset and Pre-processing

• Source: The study utilized the Webis-CLS-10 dataset (Prettenhofer and Stein), containing product reviews from Amazon in English, French, German, and Japanese.
• Selection: The research focused exclusively on English and French reviews across three categories: Music, DVDs, and Books.
• Balancing: To ensure linguistic equity, the dataset was strictly balanced. The authors identified the smallest subgroup (e.g., French negative reviews) and sampled an equal number of reviews from all other subgroups (English positive/negative, French positive) to create three distinct, balanced datasets:
  • Music: 31,880 reviews
  • DVDs: 18,716 reviews
  • Books: 65,740 reviews
• Pre-processing: Text was converted to lowercase, stop words were filtered, and SpaCy was used for Named Entity Recognition (NER) and lemmatization to reduce linguistic complexity.
• Feature Engineering: TF-IDF (Term Frequency-Inverse Document Frequency) was used to vectorize the text, balancing term frequency against document frequency to highlight context-specific words.

Models and Optimization

• Algorithms: Two conventional machine learning models were employed:
  1. Support Vector Machine (SVM): Chosen for its robustness in high-dimensional text data.
  2. Naive Bayes: Selected for its effectiveness in smaller datasets and interpretability.
• Hyperparameter Tuning: Optuna (using Bayesian optimization) was used to tune hyperparameters (e.g., penalty parameter $C$ and kernel type for SVM; smoothing parameter $\alpha$ for Naive Bayes).
  • Note: The tuning process revealed a performance plateau (~0.5039 accuracy in cross-validation trials), leading the authors to use default parameters for the final models.
• Data Split: An 80/20 train-test split was applied to ensure balanced representation in both sets.

Fairness Evaluation Framework

The study utilized the Fairlearn library to quantify bias using four specific metrics, treating "Language" (English vs. French) as the sensitive attribute:

1. Demographic Parity Difference (DPD): Measures the difference in positive classification rates between languages. (Ideal: 0).
2. Equalized Odds Difference (EOD): Measures the maximum difference in True Positive Rates (TPR) and False Positive Rates (FPR) between languages. (Ideal: 0).
3. Equalized Odds Ratio (EOR): A ratio-based metric for TPR and FPR. (Ideal: 1).
4. Demographic Parity Ratio (DPR): A ratio-based metric for selection rates. (Ideal: 1).

3. Key Results

Performance Metrics (Accuracy)

Both models exhibited a consistent bias in favor of the French language, achieving higher accuracy, precision, recall, and F1-scores on French data compared to English data across all three categories.

• SVM Performance:
  • Music: French (0.918) vs. English (0.859)
  • DVDs: French (0.864) vs. English (0.854)
  • Books: French (0.889) vs. English (0.861)
• Naive Bayes Performance:
  • Music: French (0.889) vs. English (0.838)
  • DVDs: French (0.865) vs. English (0.845)
  • Books: French (0.873) vs. English (0.837)

Fairness Metrics

The fairness evaluation revealed significant disparities, particularly in the Music category.

• SVM Fairness:
  • Demographic Parity Ratio (DPR): High equity (0.962 for Music, 0.990 for DVDs, 0.985 for Books), suggesting the models classify positive sentiments at similar rates for both languages.
  • Equalized Odds Ratio (EOR): Lower equity, especially in Music (0.528), indicating that the model's ability to correctly identify true positives vs. false positives is significantly less consistent for English than French in this domain.
• Naive Bayes Fairness:
  • Demonstrated greater disparity than SVM.
  • Music DPR: 0.813 (Significant bias).
  • Music EOR: 0.352 (Severe disparity in accuracy consistency).
  • Music DPD/EOD: Highest differences among all datasets, indicating the model is least fair when analyzing music reviews.

4. Discussion and Analysis

The authors attribute the performance gap and bias to several factors:

• Linguistic Nuance: French reviews in the dataset appeared to have clearer sentiment indicators and less ambiguity compared to English.
• Cultural Specificity in Music: The Music category showed the highest bias. The authors argue that music reviews rely heavily on slang, idioms, and cultural context. The training data may have captured French cultural expressions more effectively than English ones, or the English reviews contained a wider variety of styles that diluted the model's ability to recognize sentiment patterns consistently.
• Model Sensitivity: Naive Bayes, with its assumption of feature independence, struggled more with the complex, idiomatic nature of music reviews compared to SVM.

5. Significance and Contributions

• Bias Detection in Balanced Data: The study demonstrates that even with perfectly balanced datasets, language bias persists in conventional ML models. This challenges the assumption that data balancing alone solves fairness issues in NLP.
• Domain-Specific Bias: It highlights that bias is not uniform; it is exacerbated in domains with high cultural and linguistic nuance (like Music) compared to more formal domains (like Books).
• Metric Utility: The paper validates the use of Fairlearn metrics (specifically EOR and DPR) to uncover subtle biases that accuracy metrics alone might miss.
• Call for Diverse Evaluation: The findings advocate for the development of equitable multilingual NLP systems that go beyond English-centric training and require rigorous, domain-specific fairness testing.

6. Limitations

• Model Scope: The study is limited to SVM and Naive Bayes; it does not evaluate modern Transformer-based models (e.g., BERT, mBERT), which might behave differently.
• Domain Scope: Results are specific to product reviews (Music, DVDs, Books) and may not generalize to other text types (e.g., news, social media).
• Pre-processing: Uniform pre-processing (lemmatization, stop-word removal) was applied to both languages, potentially ignoring unique linguistic traits that could improve fairness.
• Computational Constraints: The study could not perform extensive hyperparameter tuning (GridSearch) or scalability tests due to resource limitations.

Conclusion

The paper concludes that linguistic features and dataset composition significantly influence model outcomes, exposing gaps in current multilingual NLP systems. Even when English and French are equally represented, models tend to favor French in sentiment analysis tasks, particularly in culturally rich domains like music. The authors emphasize the need for targeted interventions, comparative testing between monolingual and multilingual models, and the inclusion of diverse datasets to ensure equitable access to NLP technologies.