IndoBERT-Sentiment: Context-Conditioned Sentiment Classification for Indonesian Text

1. Problem Statement

Current sentiment analysis models for Bahasa Indonesia suffer from a fundamental limitation: they treat text in isolation, ignoring the topical context that determines sentiment polarity.

• The Core Issue: Sentiment is not an intrinsic property of words but a relationship between text and a topic. A statement like "numbers keep rising" is negative in the context of inflation but positive in the context of economic growth. Similarly, "KPK arrests a corrupt regent" is positive regarding anti-corruption efforts but contains negative keywords ("arrest," "corrupt") that confuse context-free models.
• Current State: Widely used Indonesian sentiment models (fine-tuned on product reviews like the IndoNLU SmSA dataset) fail to generalize to news and social media monitoring. They default to "neutral" for factual statements or misclassify positive news (e.g., successful law enforcement) as negative due to the presence of negative vocabulary.

2. Methodology

The authors propose IndoBERT-Sentiment, a model that conditions sentiment prediction on a specific topical context.

• Architecture:
  • Base Model: IndoBERT Large (335M parameters).
  • Input Format: [CLS] context [SEP] text [SEP].
  • Mechanism: The model receives a short topic description (context) and the target text simultaneously. The [CLS] token representation is mapped to a classification head with three output classes: Negatif (Negative), Netral (Neutral), and Positif (Positive).
• Dataset Construction:
  • Source: Reuses 31,360 context–text pairs originally created for a relevancy classification task (IndoBERT-Relevancy).
  • Coverage: 188 topics across 12 thematic domains (e.g., economy, corruption, environment).
  • Data Types: Includes formal news headlines, informal social media posts, and LLM-generated implicit references.
  • Re-labeling: The original binary relevancy labels were replaced with sentiment labels using GPT-4o-mini. The prompt instructed the LLM to determine sentiment relative to the provided context.
  • Distribution: The dataset is imbalanced (55.2% Neutral, 33.0% Negative, 11.8% Positive), reflecting real-world news/social media distributions.
• Training Protocol:
  • Trained for 5 epochs with a learning rate of $2 \times 10^{-5}$ and batch size 16.
  • Class Balancing: Inverse-frequency class weighting was applied (Positif weight: 2.834) to address the minority class.
  • Hardware: Trained on a single NVIDIA RTX 3090 GPU (~30 minutes).

3. Key Contributions

1. Transferability Demonstration: Proves that context-conditioned classification, previously successful for relevancy, effectively transfers to sentiment analysis using the same architecture and dataset (re-labeled).
2. Comprehensive Benchmarking: Conducted a head-to-head evaluation against the three most downloaded general-purpose Indonesian sentiment models on the same test set.
3. Public Release: Released two variants of the model on HuggingFace:
   • Three-class: Negatif/Netral/Positif.
   • Binary: Negatif/Positif (optimized for polarity detection).

4. Results

The model was evaluated on a held-out test set of 4,704 samples against three baselines: BERT-Indonesian-SmSA, RoBERTa-Indonesian-Sentiment, and IndoBERT-Sentiment-SmSA.

• Overall Performance:

  • IndoBERT-Sentiment (Ours): Accuracy 88.1%, F1 Macro 0.856.
  • Best Baseline: Accuracy ~62.8%, F1 Macro ~0.501.
  • Improvement: Outperforms the best baseline by 25.3 accuracy points and 35.6 F1 macro points.

• Per-Class Analysis (The "Positif Gap"):

  • Negative & Neutral: Baselines perform reasonably well (F1 ~0.60–0.71).
  • Positive: Baselines collapse, achieving F1 scores between 0.135 and 0.211 (correctly identifying <20% of positive texts).
  • Ours: Achieves an F1 of 0.791 for the Positive class (a >275% relative improvement).
  • Reasoning: Baselines rely on explicit positive adjectives (e.g., "good," "like") found in product reviews. In public discourse, positive sentiment is often expressed via domain-specific achievements (e.g., "growth," "arrested," "decreased") which require context to interpret as positive.

• Binary Variant:

  • By removing the "Neutral" class, the binary model achieved 96.06% accuracy and 0.949 F1 Macro, ideal for applications requiring clear polarity signals.

5. Significance and Implications

• Paradigm Shift: The paper argues that for social media monitoring and news analysis, sentiment is inherently context-dependent. Treating it as an isolated task leads to systematic errors.
• Efficiency: The approach demonstrates that a single dataset and architecture can serve multiple tasks (relevancy, sentiment, stance) simply by changing the labeling strategy, reducing the need for massive, task-specific data collection.
• Practical Application: The model solves the "domain mismatch" problem where models trained on product reviews fail on public discourse. It enables systems to correctly interpret complex statements (e.g., distinguishing between a "rise in prices" as bad news vs. a "rise in GDP" as good news) without manual rule engineering.
• Limitations: The model requires a topical context at inference time (adding a step to the pipeline) and relies on a dataset labeled with context-dependent ground truth, which may disadvantage context-free models in a direct comparison. However, the authors argue context-dependence is the correct standard for the target use cases.