Technical Summary: Fault of Our Stars – Behavioral Drivers of Rating–Sentiment Incongruence

Problem Statement

In the domain of Natural Language Processing (NLP) and tourism analytics, star ratings are frequently utilized as convenient "weak labels" to represent the sentiment of accompanying review text. This study challenges the assumption that numerical ratings and written reviews consistently align. The core problem addressed is sentiment–rating incongruence, where the sentiment polarity inferred from review text differs from the sentiment implied by the assigned star rating. Treating ratings as ground-truth labels without validation introduces noisy or context-biased data into sentiment analysis pipelines, potentially causing models to learn distorted patterns rather than actual linguistic sentiment. This issue is particularly critical in underrepresented tourism contexts, such as Sri Lankan attractions, where cultural expectations and venue types may influence review behavior differently than in established hotel or restaurant sectors.

Methodology

The study employs a multi-stage framework combining transformer-based NLP inference with statistical and machine learning analysis to investigate rating–text divergence.

1. Dataset and Preprocessing:

   • Source: A dataset of 16,156 reviews from Sri Lankan tourism attractions (2010–2023) covering 11 attraction types.
   • Labeling Strategy: Star ratings were grouped into three classes: Negative (1–2 stars), Neutral (3 stars), and Positive (4–5 stars).
   • Feature Engineering: Variables included review length, review delay (time between visit and posting), travel year, venue type, and reviewer tier (categorized by contribution count: Novice, Casual, Active, Expert).

2. Sentiment Model Selection:

   • To derive textual sentiment independently of ratings, four transformer-based models were evaluated on a manually labeled subset of 1,000 reviews.
   • Selection Criteria: Performance was assessed using Macro F1, Accuracy, and Weighted F1.
   • Outcome: The pre-trained cardiffnlp/twitter-roberta-base-sentiment model was selected as it achieved the best balance of classification performance and generalization (Macro F1: 0.692), avoiding overfitting risks associated with fine-tuning on small datasets.

3. Incongruence Typology:

   • Mismatches between the model-inferred text sentiment and the grouped rating class were categorized into six directional patterns:
     • Conservative Rater: Positive text, moderate rating.
     • Obligatory 5-Star: Negative/Neutral text, 5-star rating.
     • Harsh Deflator: Positive text, low rating.
     • Polite Inflator: Negative text, high rating.
     • Frustrated Neutral: Negative text, neutral rating.
     • Punitive Rater: Positive text, 1-star rating.

4. Statistical and Predictive Modeling:

   • Bivariate Analysis: Chi-square and Mann–Whitney U tests with Benjamini–Hochberg correction identified significant predictors.
   • Predictive Models: Logistic Regression (linear baseline) and Random Forest (nonlinear structure) were used to model the probability of incongruence.
   • Interpretation: SHAP (SHapley Additive exPlanations) values were applied post-hoc to the Random Forest model to interpret feature importance.

Key Results

• Prevalence: Incongruence occurs in 18.6% of the analyzed reviews (3,005 instances), indicating that roughly one in five reviews contains a mismatch between text and rating.
• Directional Patterns: The mismatches are not random noise but follow a structured typology. The Conservative Rater (38.4%) and Obligatory 5-Star (28.3%) patterns account for nearly two-thirds (66.7%) of all incongruent cases.
• Venue Type Variance: Incongruence rates vary significantly by attraction type. Museums exhibit the highest incongruence rate (26.3%), while National Parks show the lowest (12.8%). Museums are over twice as likely to be incongruent compared to National Parks.
• Reviewer Expertise: Expert reviewers (101+ contributions) are 1.97 times more likely to produce incongruent reviews than novices. Specifically, the "Conservative Rater" pattern increases from 27.4% among novices to 40.4% among experts, suggesting that experienced users calibrate their numerical scores more strictly than their written praise.
• Temporal Trends: There is a modest but consistent linear decline in incongruence over time, suggesting that recent reviews show greater alignment between text sentiment and star ratings.
• Model Performance: Both Logistic Regression (AUC ≈ 0.584) and Random Forest (AUC ≈ 0.609) demonstrated modest predictive performance, indicating that while factors like venue type and expertise are significant, a substantial portion of incongruence is driven by latent, unmeasured behavioral or contextual factors.

Significance and Claims

The paper argues that sentiment–rating incongruence is a systematic, context-dependent signal rather than random noise. Its primary contributions and implications are:

1. Re-evaluation of Weak Labels: The study demonstrates that star ratings are not interchangeable with textual sentiment. Using ratings as ground-truth labels for training sentiment analysis models without validation introduces systematic label noise, particularly in diverse tourism contexts.
2. Context-Aware Analysis: The findings highlight that rating reliability is not uniform; it is heavily influenced by venue type (e.g., museums vs. parks) and reviewer expertise. This necessitates context-aware sentiment analysis approaches that validate weak-label reliability before model training.
3. Behavioral Insights: The identification of specific directional patterns (e.g., "Conservative Rater") provides a framework for understanding how and why mismatches occur, moving beyond simple binary classifications of match/mismatch.
4. Methodological Rigor: By using transformer-based inference independent of ratings, the study offers a robust method for detecting incongruence in low-resource settings where manual labeling is scarce, challenging previous assumptions in low-resource sentiment analysis literature that rely heavily on rating-derived labels.

The authors conclude that for both NLP research and tourism analytics, treating ratings and text as distinct, complementary signals is essential for accurate analysis, and that rating-based labels should be scrutinized for domain-specific bias before being utilized as ground truth.