💬 NLP

Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media

This paper demonstrates that both supervised language models and ChatGPT can effectively perform automated stance detection on the complex, lower-resource Estonian language regarding immigration, enabling the analysis of diachronic trends in polarizing news media.

Original authors: Mark Mets, Andres Karjus, Indrek Ibrus, Maximilian Schich

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mark Mets, Andres Karjus, Indrek Ibrus, Maximilian Schich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, noisy town square where thousands of people are shouting about a single, controversial topic: Immigration. Some people are shouting "Welcome!" while others are screaming "Go away!" and many are just talking about the weather or the price of bread.

Your goal is to figure out who is saying what, and how the mood of the crowd changes over time. But there's a catch: you don't speak the language of the town (Estonian), and you don't have a team of 100 translators to read every single sentence.

This paper is about building a digital translator and mood-reader to solve this problem. Here is the story of how they did it, broken down into simple parts.

1. The Challenge: A Tiny Language and a Big Problem

Most computer programs that understand language are like giant libraries built for English. They have read millions of books. But for smaller languages like Estonian (spoken by only 1.1 million people), the library is very small.

The researchers wanted to see if they could teach a computer to understand the "stance" (the opinion) on immigration in Estonian news. This is tricky because:

The language is complex: Estonian words change their shape a lot (like adding different endings), making them hard for computers to recognize.
The topic is emotional: People use sarcasm, metaphors, and "dog whistles" (coded language) to hide their true feelings.
The data is scarce: There aren't enough pre-labeled examples to teach the computer the old-fashioned way.

2. The Experiment: Two Ways to Teach the Computer

The researchers tried two different methods to teach the computer how to sort the shouting crowd into three groups: Pro-Immigration, Anti-Immigration, and Neutral.

Method A: The "Apprentice" (Supervised Learning)

Think of this as hiring a human teacher to train a student.

The Lesson: They took 8,000 sentences from news articles and had human students read them, labeling each one as "Pro," "Anti," or "Neutral."
The Training: They fed these labeled sentences to a smart computer model (a type of AI called a Large Language Model). The model studied the patterns, like a student memorizing flashcards.
The Result: The computer became a decent apprentice. It got about 66% of the answers right. It was really good at spotting "Anti" opinions but sometimes got confused between "Neutral" and "Pro."

Method B: The "Magic 8-Ball" (Zero-Shot ChatGPT)

This was the exciting new part. Instead of training a model with thousands of examples, they just asked a powerful, pre-existing AI (ChatGPT) a simple question in plain English:

"Here is a sentence about immigration. Is the writer for it, against it, or neutral? Just give me the label."

It's like asking a genius who has read the whole internet to guess the answer without any specific homework.

The Result: Surprisingly, the "Magic 8-Ball" performed almost exactly as well as the trained apprentice (65% accuracy).
Why it matters: This means you might not need to spend months and thousands of dollars labeling data for every new language. You can just ask the AI nicely, and it might do the job.

3. The Detective Work: What Did They Find?

Once they had their best "digital detective" (the trained model), they let it read 100,000+ sentences from two very different Estonian news sources over seven years (2015–2022):

Source A (Ekspress Grupp): A mainstream, general-interest news group (like a balanced newspaper).
Source B (Uued Uudised): A right-wing populist portal (like a political activist's blog).

The Findings:

The Split: As expected, the populist source was almost always "Anti-Immigration," while the mainstream source was mostly "Neutral."
The Shifts: The computer tracked how the mood changed during big world events:
- 2015-2016 (Migration Crisis): Both sources talked more about it, but the populist source got louder and more negative.
- 2018-2019 (UN Pact & Elections): The "Anti" stance spiked in the populist source right before elections, showing how politicians use these topics to rally voters.
- 2022 (Ukraine War): This was the most interesting twist. When Russia invaded Ukraine, the mainstream source suddenly became much more "Pro-Immigration" (because Ukrainians were fleeing to Estonia). The populist source also shifted slightly, but they remained more skeptical.

4. The Limitations: Why It's Not Perfect

The researchers were honest about the flaws.

Context is King: Sometimes a single sentence is a trap. If a sentence says, "They criticize racism," the computer might think it's "Pro-Immigration." But if the sentence is actually quoting a villain saying that, the computer gets it wrong. It's like hearing a joke without knowing who is telling it.
Human Bias: The humans who labeled the training data had their own biases. If the teacher is biased, the student (the AI) will be too.
The "Black Box": With tools like ChatGPT, we don't always know how it got the answer, which makes it hard to trust 100%.

The Big Takeaway

This paper is a proof of concept. It shows that even for a small, difficult language and a messy, emotional topic, AI can be a useful tool for media monitoring.

It's like giving a researcher a pair of super-vision glasses. They can't read every single word in a newspaper, but with these glasses, they can quickly scan thousands of articles to see the big picture: Is the tone getting more angry? Is the topic shifting? Are the two sides of the political spectrum drifting further apart?

And the best news? You might not need a massive team of humans to build the glasses anymore; a simple prompt to a smart AI might be enough to get you started.

1. Problem Statement

The paper addresses the challenge of automated stance detection in low-resource, morphologically complex languages (specifically Estonian) regarding socio-politically complex and polarizing topics (immigration).

The Gap: Traditional supervised machine learning approaches require large, annotated training datasets, which are often unavailable for smaller languages or specific niche topics.
The Complexity: Stance detection differs from sentiment analysis; it requires identifying a specific position (For, Against, Neutral) toward a specific target (immigration), rather than general positivity/negativity. This is complicated by sarcasm, implicit opinions, third-party references, and "dog-whistle" rhetoric common in polarizing media.
The Goal: To evaluate the feasibility of using Large Language Models (LLMs), including both fine-tuned supervised models and zero-shot generative models (ChatGPT), to detect stances in Estonian news media without relying on massive, pre-existing labeled datasets.

2. Methodology

Data Collection and Preprocessing

Corpus: The study analyzed news articles from 2015 to April 2022 from two distinct sources:
- Mainstream: Ekspress Grupp (including the portal Delfi and various newspapers), representing a broad, centrist perspective.
- Radical-Right Populist: Uued Uudised ("New News"), an online portal aligned with the radical-right EKRE party.
Unit of Analysis: Sentences were chosen over paragraphs or articles to mitigate the issue of multiple stances within a single text and to fit BERT-based model input limits (512 tokens).
Extraction: 8,000 sentences were extracted using a custom dictionary of keywords and regular expressions covering migration, refugees, foreign workers, race, and political opposition terms (e.g., "multiculturalism," "globalists"). A negative filter removed irrelevant topics (e.g., bird migration).
Dataset Size: The final dataset consisted of 7,345 annotated sentences after filtering out ambiguous or non-topical entries.

Annotation Process

Human Annotation: Two graduate students annotated sentences on a 1–5 scale (1=Against, 5=Supportive).
Classes: Ratings were collapsed into four classes: Against (1-2), Neutral (3), Supportive (4-5), and Ambiguous (excluded from training).
Inter-rater Reliability: Substantial agreement was found ( $\kappa \approx 0.66-0.69$ ), with very high agreement ( $\kappa = 0.97$ ) when collapsing to binary Pro/Con classes.

Model Architecture and Training

The authors compared several approaches:

Supervised Fine-Tuning: Several pre-trained Transformer models were fine-tuned on the annotated Estonian dataset:
- Est-RoBERTa (Monolingual Estonian)
- EstBERT (Monolingual Estonian)
- XLM-RoBERTa (Multilingual)
- mBERT (Multilingual)
- Baseline: A model fine-tuned on a general Estonian sentiment analysis dataset (Pajupuu et al., 2016) to test if sentiment transfer works for stance.
Zero-Shot Classification: ChatGPT (GPT-3.5) was used as an instructable zero-shot classifier. The model was prompted with natural language instructions defining the stance classes without any fine-tuning on the specific dataset.
Evaluation: 5-fold cross-validation was used. Performance was measured using F1-macro scores, precision, and recall.

3. Key Contributions

Proof of Concept for Low-Resource Languages: Demonstrates that automated stance detection is feasible in Estonian (a low-resource language) for a highly complex topic, achieving acceptable accuracy without massive proprietary datasets.
Zero-Shot vs. Supervised Comparison: Provides empirical evidence that ChatGPT (zero-shot) performs nearly on par with the best fine-tuned supervised model (F1 0.65 vs. 0.66), suggesting a cheaper, faster alternative for media monitoring.
Public Dataset: The authors released a 7,345-sentence annotated dataset for Estonian stance detection, a valuable resource for the NLP and media studies communities.
Diachronic Media Analysis: Applied the best-performing model to a large corpus ( $n=106,539$ sentences) to map stance trends over 7 years, correlating them with real-world events (e.g., 2015 migration crisis, 2019 elections, 2022 Ukraine invasion).

4. Results

Classification Performance

Best Model: Est-RoBERTa achieved the highest F1-macro score of 0.66 (Precision: 0.65, Recall: 0.68).
Zero-Shot Performance: ChatGPT achieved an F1-macro of 0.65, statistically comparable to the best fine-tuned model.
Class Imbalance: All models performed significantly better at detecting "Against" stances (F1 ~0.74) than "Supportive" stances (F1 ~0.55).
Confusion Patterns: The primary errors occurred between Neutral and the extreme classes (Pro/Con), rather than between the two extremes. This is considered a "safer" error profile than confusing Pro and Con directly.
Sentiment Baseline: Using a general sentiment dataset for fine-tuning yielded poor results (F1 0.49), confirming that stance detection requires topic-specific training or zero-shot prompting.

Diachronic Analysis Findings

Polarization: Uued Uudised consistently maintained a strong "Against" stance, while Ekspress Grupp remained predominantly Neutral.
Event Correlation:
- 2018-2019 (UN Migration Pact & Elections): Both outlets showed a spike in "Against" rhetoric, indicating the politicization of the topic.
- 2020 (Pandemic): A shift occurred where radical-right focus moved from anti-immigration to anti-government.
- 2022 (Ukraine Invasion): Ekspress Grupp showed a significant increase in Supportive stances toward immigrants. Uued Uudised showed a smaller increase in support, reflecting the complex rhetoric of the radical right regarding Ukrainian refugees vs. other migrants.
Cosine Similarity: Sentence embeddings showed spikes in similarity between the two outlets during major events (late 2018, 2022), suggesting they were discussing the same events, albeit with different stances.

5. Significance and Limitations

Significance:

Methodological Shift: Validates that generative AI (LLMs) can replace expensive, time-consuming human annotation pipelines for stance detection in low-resource languages, democratizing media monitoring.
Political Insight: Offers a quantitative method to track media polarization and the evolution of radical-right rhetoric in Estonia, linking linguistic shifts to geopolitical events.
Scalability: The approach is scalable to other complex topics and languages where annotated data is scarce.

Limitations:

Annotation Bias: Human annotators rely on cultural context and prior knowledge, which introduces subjectivity. The distinction between "Neutral" and "Ambiguous" was particularly difficult.
Unit of Analysis: Sentence-level analysis misses cross-sentence context. Paragraph-level analysis might capture nuance better but introduces multi-stance confusion.
Model Replicability: Zero-shot results with ChatGPT are difficult to replicate exactly due to the stochastic nature of the model and its continuous updates (closed-source).
Error Types: The models struggle with sarcasm, irony, and sentences referencing third-party opinions (e.g., "They say X is bad" vs. "X is bad").

Conclusion:
The paper successfully demonstrates that automated stance detection is a viable tool for analyzing polarizing media in small languages. It highlights that while supervised fine-tuning remains robust, zero-shot prompting with models like ChatGPT offers a highly competitive, cost-effective alternative, particularly for rapid media monitoring and trend analysis.