Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale

Technical Summary: Creation of the Estonian Subjectivity Dataset

Problem Statement
Subjectivity analysis is a foundational task in Natural Language Processing (NLP) for understanding human opinions, attitudes, and perspectives. While existing datasets often rely on binary classification (objective vs. subjective) or sentence-level analysis, human perception of subjectivity is inherently gradual and exists on a continuum. Furthermore, there is a lack of resources for low-resource languages; specifically, no subjectivity dataset existed for Estonian prior to this work. The challenge lies in creating a reliable dataset that captures the nuance of subjectivity on a continuous scale while addressing the inherent variability in human annotation and exploring the feasibility of automating this process using Large Language Models (LLMs).

Methodology
The authors constructed the Estonian Subjectivity Dataset, comprising 1,000 documents rated on a continuous scale from 0 (fully objective) to 100 (fully subjective).

• Data Collection: The dataset includes 300 journalistic texts (150 news articles, 150 opinion pieces) and 700 randomly selected web texts from the 2021 Estonian National Corpus. Texts were constrained to 100–6,000 characters to ensure meaningful analysis while maintaining annotator focus. The corpus covers diverse genres, including advertising, social media, legal texts, and literature.
• Annotation Protocol:
  • Pilot Study: A preliminary study with 60 texts validated the continuous scale and the selection of sources (ranging from neutral public broadcasting to opinionated alternative media).
  • Main Annotation: Three annotators (A1, A2, A3) rated the full 1,000-text dataset using a sliding scale in a survey environment. They also provided confidence ratings ("not at all certain" to "entirely certain").
  • Re-annotation: To investigate high inter-annotator disagreement, a subset of 250 texts (selected for extreme score divergences) was re-annotated by two of the original annotators after a four-month interval.
  • Fourth Annotator: A fourth annotator (A4), who participated in the pilot, was recruited to annotate the full dataset to improve score aggregation reliability.
• LLM Experiment: The authors utilized GPT-5 to annotate the entire 1,000-text dataset. The model was prompted to assign a score (0–100) and provide a rationale. To assess consistency, the dataset was processed in three separate batches, and the scores were averaged.

Key Contributions

1. First Estonian Subjectivity Dataset: The creation of a document-level subjectivity dataset for Estonian, annotated on a continuous 0–100 scale rather than a categorical binary.
2. Analysis of Annotation Variability: A detailed examination of inter-annotator agreement, revealing that while correlations are moderate, they are statistically significant. The study identifies that text length and genre influence annotation consistency differently.
3. Contextual Effects: The paper documents the influence of annotation context (the sequence of texts presented) on human scoring, noting that annotators may subconsciously adjust their internal scales based on preceding texts.
4. Human-LLM Comparison: An empirical comparison between human annotations and GPT-5 outputs, highlighting specific systematic divergences in how humans and models interpret subjectivity cues.

Results

• Inter-Annotator Agreement: Initial correlations between the three primary annotators were moderate (Pearson's r ranging from 0.525 to 0.627). Re-annotation of the high-disagreement subset improved agreement between the two re-annotating participants to r = 0.678. The fourth annotator showed consistent correlations (0.627–0.675) with the initial group.
• Factors Influencing Scores:
  • Text Length: Longer texts were generally rated as more subjective, though this correlation varied significantly by genre (e.g., strong in legal texts, negligible in news).
  • Context: Annotators reported that scoring a series of subjective texts made them more sensitive to objectivity in subsequent texts, suggesting sequence effects.
  • Confidence: Annotator confidence ratings did not correlate strongly with the degree of disagreement between annotators.
• LLM Performance: GPT-5 demonstrated high internal consistency across three runs (Pearson's r ≈ 0.98). However, when compared to humans, the model showed distinct patterns:
  • Score Range: GPT-5 rarely assigned scores above 98, whereas humans utilized the full 0–100 range.
  • Quotations: GPT-5 tended to rate texts containing direct quotes (common in news) as more subjective than humans, likely interpreting the quoted content as part of the text's subjectivity rather than a reported fact.
  • Tone vs. Content: Humans rated texts with colloquial language, slang, and emoticons (e.g., forum posts) as highly subjective. GPT-5 focused more on the factual content, assigning lower subjectivity scores to these texts.

Significance and Claims
The paper claims that the created dataset is a valid and useful resource for linguistic research and the development of automatic subjectivity detection systems. The authors assert that:

• Human annotators can consistently use a continuous numeric scale for subjectivity, despite moderate inter-annotator disagreement.
• LLM-based automatic subjectivity scoring is feasible and produces plausible scores that correlate moderately with human ratings.
• However, LLMs are not an interchangeable alternative to human annotation. The study concludes that LLMs rely on different cues (e.g., prioritizing content over tone) compared to humans. Therefore, while LLMs may complement human efforts in applications requiring speed and automation, they cannot yet replace human annotators in tasks requiring nuanced, human-like understanding of subjectivity, particularly in linguistic or interpretive research.

The authors emphasize that future work should focus on expanding the dataset with additional human annotations to better understand context effects and on refining prompt engineering to reduce human-LLM discrepancies.