💻 computer science

How Annotator Agreement Affects Sentiment Classification Performance for Indonesian Coastal Tourism Reviews

This study demonstrates that training sentiment classifiers on a larger, majority-vote dataset of Indonesian coastal tourism reviews yields higher performance than a strictly unanimous-consensus dataset, though the results suggest that increased data volume and class balance, rather than the retention of ambiguous instances alone, may be the primary drivers of improvement while IndoBERT effectively reduces systematic misclassification of neutral reviews.

Original authors: Sandy Kurniawan, Henri Tantyoko, Khadijah Khadijah, Helmie Arif Wibawa

Published 2026-08-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sandy Kurniawan, Henri Tantyoko, Khadijah Khadijah, Helmie Arif Wibawa

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, the voice of the traveler has become a massive, unstructured library of human opinion. Millions of people write reviews about beaches, hotels, and restaurants, pouring their hopes, disappointments, and observations into text that is often informal, slang-filled, and deeply nuanced. For businesses and researchers, the challenge is to make sense of this flood. They need to know not just if a review is good or bad, but exactly how the writer feels. This is the domain of sentiment analysis, a branch of computer science that teaches machines to read emotion. For years, the standard approach has been to have humans label these reviews with simple categories like "positive," "negative," or "neutral," and then train computers to recognize those patterns. However, human judgment is rarely perfect. When three people read the same review, they might disagree on whether it is truly neutral or slightly negative. This disagreement creates a puzzle for scientists: should they throw away the confusing reviews to keep only the ones everyone agrees on, or should they keep the messy, disputed ones to give the computer more examples to learn from?

A team of researchers at Diponegoro University in Indonesia set out to solve this specific puzzle using reviews of coastal tourism destinations. They focused on the Indonesian language, which is rich with informal expressions and regional variations, making it a particularly difficult test case for computers. The researchers gathered hundreds of thousands of public reviews from Google Maps covering beaches across the archipelago. After cleaning the data and filtering out duplicates or non-Indonesian text, they selected a balanced sample of roughly 22,000 reviews to be read by three human annotators. Each person independently decided if a review was positive, negative, or neutral. This process revealed a natural reality: the humans did not always agree. Some reviews were so clear that all three annotators chose the same label, while others were ambiguous enough that two agreed and one disagreed, or where all three chose different labels.

The researchers then split their data into two distinct groups to see how this disagreement affected the computer's learning. The first group, which they called the "unanimous" set, contained only the reviews where all three humans agreed. This was a smaller, cleaner collection of 15,527 reviews, but it was missing the tricky, ambiguous cases. The second group, the "majority" set, kept the unanimous reviews but also added the ones where two out of three humans agreed. This larger set contained 21,235 reviews and included many more examples of the neutral, mixed-sentiment, or confusing text that often trips up computers. They then trained three different types of computer programs on these two groups. Two of the programs used older, traditional methods that look at word frequencies, while the third used a modern, advanced system called IndoBERT, which is designed to understand the context and relationships between words in the Indonesian language, much like a human reader does.

The results were clear and somewhat surprising. Contrary to the idea that a computer learns best only from perfectly agreed-upon examples, every single program performed better when trained on the larger, messier group that included the disputed reviews. The most advanced system, IndoBERT, achieved a higher score when it learned from the majority-vote group compared to the unanimous group. This suggests that by throwing away the ambiguous reviews to ensure perfect agreement, the researchers were actually throwing away valuable lessons. The disputed reviews often contained the complex, mixed feelings that real travelers express, and having more of these examples helped the computer understand the full range of human emotion. The study found that the biggest improvement came in the ability to correctly identify "neutral" reviews. When the computer was trained only on the clean, unanimous data, it struggled to distinguish between a truly neutral review and one that was slightly positive or negative. But when it was allowed to study the majority-vote data, which had many more examples of these neutral cases, it became much better at recognizing them.

However, the researchers were careful to note that this improvement might not be solely due to the disagreement itself. The larger group simply had more data and a better balance of neutral examples, which are naturally harder to find than clear-cut positive or negative ones. The study did not prove that disagreement is good in itself, but rather that filtering it out removes too much useful information. The computer still struggled with certain types of complex writing, such as reviews that praised one part of a beach while criticizing another, or those that used subtle, indirect language to express disappointment. These difficult cases remained a challenge even for the most advanced model. Ultimately, the work suggests that in the messy world of human language, a little bit of disagreement in the training data might be necessary to teach a computer to truly understand the nuance of a traveler's voice.

Technical Summary: Annotator Agreement and Sentiment Classification in Indonesian Coastal Tourism

Problem Statement
The processing of User-Generated Content (UGC) for regional tourism, specifically Indonesian coastal reviews, is hindered by linguistic complexity, ambiguity, and the subjective nature of sentiment. While deep contextual models like IndoBERT offer superior semantic comprehension compared to traditional baselines, their effectiveness is fundamentally bounded by the quality of annotated training data. A central methodological challenge in subjective NLP tasks is the treatment of annotator disagreement. Conventional strategies often employ majority voting to resolve disputes, discarding information about the disagreement itself, or filter data to retain only unanimous consensus. This creates a trade-off: filtering for high consensus may improve label consistency but removes linguistically ambiguous and informative examples, while retaining majority-voted instances increases dataset size and diversity but introduces potential label noise. The specific impact of these annotation-agreement conditions on downstream sentiment classification performance, particularly within the Indonesian domain, remains underexplored.

Methodology
The study constructs a dataset of 21,779 Indonesian coastal tourism reviews scraped from Google Maps, covering regions from Sumatra to Papua. The data underwent stratified undersampling to balance original star ratings, followed by manual annotation by three independent human annotators assigning labels of positive, negative, or neutral.

The researchers organized the data into two distinct experimental training tracks to isolate the effects of annotation agreement:

Track A (Unanimous Consensus): Contains only reviews where all three annotators assigned the same label ( $n=15,527$ ). This subset is heavily skewed toward polarized sentiments, with neutral instances comprising only 9.48% of the data.
Track B (Majority Vote): Retains reviews resolved by a two-to-one majority agreement ( $n=21,235$ ). This subset includes all unanimous instances plus those with resolved disagreement, resulting in a significantly higher proportion of neutral labels (20.08%) and a larger vocabulary.

Three models were trained and evaluated under both conditions using an identical, held-out test set of 300 balanced reviews (100 per class):

Linear Support Vector Classifier (SVM) with TF-IDF features (unigrams and bigrams).
Multinomial Naive Bayes with TF-IDF features.
IndoBERT (indobenchmark/indobert-base-p1), a Transformer-based model pretrained on the Indonesian Indo4B corpus.

Preprocessing pipelines differed by model family: traditional models utilized stemming and stopword removal (retaining negation words), while IndoBERT utilized a less aggressive pipeline preserving original morphological forms and punctuation to maintain contextual integrity.

Key Results
Contrary to the hypothesis that higher annotation consensus yields better performance, all three classifiers achieved higher Accuracy and Macro F1 scores when trained on Track B (Majority Vote) compared to Track A (Unanimous Consensus).

IndoBERT: Achieved a Macro F1 of 0.9327 on Track B versus 0.9255 on Track A.
SVM: Improved from a Macro F1 of 0.7223 (Track A) to 0.7552 (Track B).
Naive Bayes: Showed the most significant relative gain, moving from 0.5084 (Track A) to 0.6433 (Track B).

Error Analysis and "Neutrality Bleed"
Confusion matrix analysis revealed that the neutral class was the primary source of error, particularly for traditional TF-IDF-based models. A phenomenon termed "Neutrality Bleed" was observed, where traditional classifiers systematically misclassified neutral reviews as polarized (positive or negative).

On Track A, Naive Bayes failed to identify any neutral test instances (Recall = 0.00), misclassifying them as negative or positive.
IndoBERT demonstrated superior robustness, achieving a neutral recall of 0.83 on Track A and 0.86 on Track B, significantly outperforming the baselines in distinguishing neutral from polarized sentiments.

Qualitative analysis identified specific linguistic challenges that remained difficult even for IndoBERT, including:

Spatially conflicting sentiment: Reviews evaluating different parts of a destination with opposing polarities (e.g., trash on the west side vs. great waves on the north).
Multi-aspect mixed evaluations: Long reviews containing multiple positive and negative aspects that aggregate to a neutral rating but are often misclassified as negative due to the cumulative weight of negative cues.
Indirect criticism: Subtle expressions of dissatisfaction (e.g., "it is a pity") that models sometimes misinterpret as neutral.

Significance and Claims
The paper claims that, within the context of this specific dataset and experimental setup, retaining majority-resolved instances (Track B) yields better predictive performance than restricting training to unanimous consensus (Track A). The authors attribute this improvement to the increased training-set size, the broader vocabulary, and, crucially, the enhanced representation of the neutral class in Track B.

However, the authors explicitly state that these findings cannot be attributed solely to annotator agreement. Because Track B is a superset of Track A with different class distributions and sample sizes, the observed performance gains may equally reflect the benefits of increased data volume and class balance. Consequently, the study does not claim that annotator disagreement is inherently beneficial; rather, it highlights a trade-off where filtering for consensus removes informative, ambiguous examples that are critical for training robust models, particularly for the neutral class.

The study concludes that while IndoBERT substantially mitigates errors associated with negation and mixed-sentiment constructions compared to traditional baselines, domain-specific challenges such as multi-aspect evaluations and regional expressions persist. The authors call for future controlled experiments that hold training-set size and class distribution constant to isolate the specific contribution of annotator agreement, and suggest moving toward Aspect-Based Sentiment Analysis (ABSA) to better handle reviews with conflicting polarities.

Technical Summary: Annotator Agreement and Sentiment Classification in Indonesian Coastal Tourism

More like this