100,000+ Movie Reviews from Kazakhstan: Russian, Kazakh, and Code-Switched Texts

Technical Summary: 100,000+ Movie Reviews from Kazakhstan

Problem Statement
Publicly available datasets for sentiment analysis in Kazakh remain scarce, limiting reproducible research in this under-resourced language. Furthermore, Kazakhstan presents a unique multilingual challenge where user-generated content is predominantly in Russian, yet includes Kazakh and frequent code-switching (Kazakh-Russian). Existing benchmarks often lack long temporal spans or fail to capture the specific sociolinguistic nuances of regional varieties (e.g., Kazakhstani Russian). Additionally, fine-grained sentiment prediction is often compromised by label leakage, where users explicitly state ratings within the review text, allowing models to bypass genuine sentiment inference.

Methodology
The authors constructed a new corpus of 100,502 movie reviews collected from kino.kz, spanning 25 years (2001–2025) and covering 4,943 unique titles. The dataset includes Russian, Kazakh, and code-switched texts.

• Annotation: Reviews were manually annotated for language and sentiment polarity (Negative, Neutral, Positive). A subset of 11,309 reviews containing explicit user ratings (0–10 scale) was extracted and normalized. To ensure annotation reliability with a single annotator, GPT-4.1 was used as a compensatory measure, achieving 89.54% accuracy and substantial agreement (Cohen's κ = 0.78) against human labels.
• Tasks: Two supervised classification tasks were defined:
  1. Polarity Classification (PC): Three-way classification (Negative/Neutral/Positive).
  2. Score Classification (SC): Five-class classification based on user ratings, grouped into bins (0–2, 3–4, 5–6, 7–8, 9–10) to address severe class imbalance.
• Data Leakage Control: For the SC task, explicit rating mentions (e.g., "9 out of 10") were replaced with a placeholder token (scoretoken) to prevent models from relying on direct lexical cues rather than inferring sentiment from context.
• Models: The study benchmarked classical baselines (Linear SVM, Logistic Regression, Naïve Bayes using BoW/TF-IDF) against multilingual transformer models (mBERT, XLM-RoBERTa, RemBERT).
• Evaluation: Data was split at the movie level (80/10/10) to prevent topical leakage. Performance was measured using Accuracy, Precision, Recall, Macro-averaged F1, and Cohen's κ, with Macro-averaged F1 prioritized due to class imbalance.

Key Results

• Polarity Classification: Transformer models consistently outperformed classical baselines. RemBERT achieved the highest performance with a test set Macro-averaged F1 of 0.82 (κ = 0.88), followed by XLM-RoBERTa (0.81) and mBERT (0.74). Classical SVM reached an F1 of 0.73.
  • Class Performance: Models performed strongly on Positive (F1=0.97) and Negative (F1=0.94) classes but struggled with the Neutral class (F1=0.56), which is rare (approx. 4–5% of data) and often expressed implicitly.
  • Language Performance: Performance correlated with data volume. Russian reviews yielded the highest F1 (0.81), while Kazakh (0.77) and code-switched (0.76) reviews showed slightly lower but respectable results.
• Score Classification: Under leakage-controlled evaluation, performance dropped significantly across all models, with Macro-averaged F1 scores ranging between 0.50 and 0.55.
  • RemBERT and SVM achieved the highest F1 (0.54–0.55).
  • The gap between transformers and classical baselines narrowed, suggesting that once explicit rating cues are masked, fine-grained score prediction relies on subtle features that deep models do not drastically outperform linear models on in this specific setting.
  • Models performed well on the most frequent high-score bin (Score 5, F1=0.88) but failed on intermediate bins (e.g., Score 2, F1=0.09) due to data sparsity.

Significance and Claims
The paper claims the dataset is a significant resource for the NLP community for three primary reasons:

1. Temporal Depth: The 25-year span allows for tracing shifts in audience attitudes toward social phenomena and the evolving role of the Kazakh language in digital discourse (e.g., the shift from requests to speak Kazakh to confident usage).
2. Multilingual Diversity: It captures naturally occurring Kazakh-Russian code-switching and "Kazakhstani Russian" varieties, including culturally specific entities and calqued constructions, providing a realistic testbed for regional language variation.
3. Diagnostic Benchmark: The experiments highlight specific dataset difficulties often missed in cleaner benchmarks: the rarity and ambiguity of neutral polarity, and the susceptibility of fine-grained score prediction to label leakage.

The authors position this work not as a definitive solution to all sentiment challenges, but as a reproducible baseline and diagnostic tool for future research on multilingual modeling, code-switching, and robust sentiment inference in real-world, culturally grounded text from Kazakhstan. The dataset, annotation guidelines, and trained models are released publicly to support these efforts.