Contextualized Prompting For Stance Detection On Social Media

Technical Summary: Contextualized Prompting For Stance Detection On Social Media

Problem Statement
Stance detection on social media, particularly Twitter, is inherently challenging due to the short, noisy, and highly context-dependent nature of user-generated content. While Large Language Models (LLMs) have demonstrated strong zero-shot and few-shot generalization capabilities, standard prompting setups typically classify tweets in isolation. This approach often fails to resolve ambiguous expressions where a single post lacks sufficient information to determine the author's viewpoint toward a specific target (e.g., a person, organization, or topic). Although prior supervised learning research suggests that incorporating metadata (such as user biographies and historical tweets) improves performance, it remains unclear whether these benefits translate to zero-shot and few-shot prompting scenarios, where LLMs are highly sensitive to prompt design and may struggle to distinguish between relevant and irrelevant context.

Methodology
The authors systematically investigate the impact of integrating various contextual features into zero-shot and few-shot prompting for stance detection. The study employs a diverse set of four reasoning-based, instruction-tuned LLMs (GPT-OSS-20B, Qwen2.5-32B, Gemma3-27B, and Ministral3-14B) across four benchmark datasets: semeval2016t6, covid19-glandt, wtwt (mergers and acquisitions), and a newly introduced high-quality German dataset, covid19-de.

The study evaluates three categories of contextual features:

1. Real-world Metadata: Retrieved via the Twitter API, including user Usernames, User Biographies, and Other Tweets (the ten most similar tweets by the same user, selected via Sentence-BERT cosine similarity).
2. Derived Metadata: Author classifications (e.g., citizen, politician, scientist) and fine-grained author groups derived from biographies.
3. LLM-Generated Context: Synthetic data generated using GPT-4, specifically Target Descriptions (a short description of the target entity) and Tweet-Specific Context (a description of the target in the specific context of the tweet).

The experimental setup compares standard zero-shot and few-shot baselines against prompts augmented with these features. The authors also evaluate Chain-of-Thought (CoT) prompting and an "LLM-as-a-Judge" approach where the model re-evaluates its initial prediction. Performance is measured using macro-averaged F1 scores, averaged across three random seeds.

Key Contributions

1. Systematic Evaluation of Context: The paper provides a comprehensive analysis of how different types of context (real, derived, and generated) affect LLM performance in stance detection, moving beyond the static knowledge bases used in previous work.
2. New Dataset: The authors release covid19-de, a new high-quality German Twitter stance detection dataset (n=3,750) focusing on governmental measures during the COVID-19 pandemic, annotated with substantial agreement (Krippendorff $\alpha > 0.67$).
3. Empirical Findings on Context Utility: The study challenges the assumption that more context is always better. It demonstrates that while LLM-generated target descriptions consistently improve performance, other contextual features (like user biographies or other tweets) yield mixed or even detrimental results.
4. Qualitative Analysis of Failure Modes: The authors identify that LLMs struggle to distinguish between informative and irrelevant context. Specifically, adding "Other Tweets" or "Tweet-Specific Context" often leads to incorrect predictions, particularly by biasing the model toward non-neutral stances (favor/against) even when the correct label is neutral.

Results

• Target Descriptions: LLM-generated target descriptions were the most robust feature, consistently improving accuracy across almost all datasets and models (e.g., +5.10pp on semeval2016t6 zero-shot). This suggests that explicit target information helps resolve implicit references in short texts.
• Other Tweets: Contrary to findings in supervised learning, providing other tweets from the same user generally impaired performance or showed no significant benefit. The increased input length appeared to distract the model, and the added context often induced a bias toward opinionated stances, degrading performance on neutral instances.
• Tweet-Specific Context: Surprisingly, LLM-generated tweet-specific context had a negative effect in every setting, likely due to the model incorporating stance-bearing words from the generated context that conflicted with the actual tweet content.
• Baselines: CoT prompting generally underperformed compared to standard zero-shot baselines on short-text tasks without additional context. The "LLM-as-a-Judge" approach was competitive, showing particular strength on the imbalanced covid19-de dataset by correcting predictions toward the majority neutral class.
• Author Metadata: Semi-automatic author classifications (e.g., politician vs. citizen) did not significantly outperform the no-context baseline, suggesting that such metadata does not translate into gains for zero-shot/few-shot LLMs.

Significance and Claims
The paper claims that while integrating contextual information holds promise for improving stance detection in noisy real-world settings, the benefits are highly conditional. The primary significance lies in highlighting the limitations of current prompting strategies: LLMs lack the robustness to effectively filter out irrelevant or conflicting context. The authors argue that simply adding more data to the prompt is insufficient; future research must focus on developing methods to differentiate between useful and noisy context, particularly for subjective tasks where relevance is not objectively defined. The work extends the line of research on contextual subjectivity to stance detection, demonstrating that the "more context is better" heuristic does not universally apply to LLM prompting.