Language-Specific Sentiment Polarity Biases in Encoder and Large Language Model Classification of Product Reviews

Technical Summary: Language-Specific Sentiment Polarity Biases in Encoder and Large Language Model Classification of Product Reviews

Problem Statement
While sentiment analysis systems are widely deployed for cross-lingual opinion mining, their reliability is often assumed to be consistent across sentiment classes (positive vs. negative). However, prior research suggests that models may exhibit polarity biases—systematic differences in accuracy between positive and negative classifications. The specific nature of these biases remains unclear regarding how they interact with different model architectures (encoder-only vs. decoder-based Large Language Models) and distinct linguistic/cultural contexts. This study investigates whether sentiment classification accuracy differs significantly between positive and negative reviews, and whether these biases are specific to both the model architecture and the target language, particularly in typologically distinct languages like French and Japanese.

Methodology
The study utilized the Amazon Multilingual Reviews Corpus, extracting a balanced dataset of 500 reviews per language (250 positive, 250 negative) for French and Japanese. Positive reviews were defined as 4–5 star ratings, and negative reviews as 1–2 star ratings.

Four models were evaluated:

1. Encoder Model: mDeBERTa-v3 (a multilingual encoder fine-tuned on MultiNLI and XNLI), utilizing a zero-shot Natural Language Inference (NLI) approach where reviews were evaluated against hypothesis templates ("The sentiment is positive/negative").
2. Large Language Models (LLMs): Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, prompted to classify sentiment with a single-word response.

Analysis and Metrics:

• Statistical Testing: Chi-square tests of independence (with Yates' continuity correction) were used to determine if accuracy differences between positive and negative classes were statistically significant within each model-language pair.
• Effect Size: Cramér's V was calculated to measure the magnitude of association.
• Error Analysis: Confusion matrices were analyzed to distinguish between False Negatives (positive reviews misclassified as negative) and False Positives (negative reviews misclassified as positive).
• Qualitative Review: Misclassified reviews were manually analyzed to identify linguistic features (e.g., indirect criticism, mixed sentiment markers) contributing to errors.
• Ensemble Potential: Inter-model agreement and error overlap were calculated to assess the feasibility of combining architectures.

Key Results
The study confirmed that sentiment polarity biases are both architecture-specific and language-specific:

• French Results (Negative Bias in LLMs):

  • All three LLMs exhibited a statistically significant negative bias, classifying negative reviews with higher accuracy (99.2%) than positive reviews (91.6%–94.4%).
  • Error analysis revealed LLMs were 7–10 times more likely to misclassify positive reviews as negative (False Negatives), often due to over-weighting explicit negative markers (e.g., "mais," "problème") in mixed-sentiment reviews.
  • mDeBERTa showed no significant bias in French, performing equally well on both classes (~92%).

• Japanese Results (Positive Bias in Encoder):

  • mDeBERTa exhibited a statistically significant positive bias, achieving 92.4% accuracy on positive reviews versus 84.4% on negative reviews.
  • The encoder model struggled with indirect criticism common in Japanese high-context communication (e.g., softening negative feedback with positive framing), leading to a high rate of False Positives (39 vs. 19).
  • Conversely, the LLMs showed no significant bias in Japanese, maintaining balanced accuracy (~95–97%) across both classes. They successfully parsed implicit negative sentiment that the encoder missed.

• Ensemble Findings:

  • LLMs showed high agreement (98.2%–98.5%) on correct classifications but low error overlap with mDeBERTa (23.7%–24.7%).
  • This suggests the architectures fail on different types of reviews, implying that an ensemble approach (e.g., stacking or routing) could achieve higher accuracy (98.0%) than any single model.

Significance and Claims
The paper claims that relying solely on overall accuracy is insufficient for deploying multilingual sentiment systems. The findings demonstrate that:

1. Bias is Context-Dependent: Polarity biases are not inherent solely to the model architecture but emerge from the interaction between architecture and language-specific communication styles (e.g., directness in French vs. indirectness in Japanese).
2. Architectural Trade-offs: Encoder models (mDeBERTa) may struggle with high-context, indirect negative expressions, while LLMs may over-react to explicit negative markers in mixed-sentiment contexts.
3. Practical Implications: Deploying a single model type globally can introduce systematic errors. For instance, using only LLMs for French analysis could undercount positive sentiment by 5–8 percentage points, while using only mDeBERTa for Japanese could miss a significant portion of customer complaints.
4. Recommendation: The authors advocate for evaluating sentiment systems on a per-sentiment-class basis rather than aggregate accuracy and suggest that ensemble methods leveraging the complementary error patterns of encoders and LLMs could mitigate these biases.

Limitations Acknowledged
The authors note that the study is limited to two languages (French and Japanese) and the domain of product reviews. They caution that patterns may differ in other languages (particularly other high-context Asian languages) or domains (social media, news). Additionally, the qualitative error analysis was exploratory and lacked formal inter-rater reliability assessment.