How people talk about each other: Modeling Generalized Intergroup Bias and Emotion

1. Problem Statement

Current Natural Language Processing (NLP) research on bias primarily focuses on identifying negative or pejorative language directed at specific demographic groups (e.g., race, gender). However, this approach is limited because:

• It requires a clear definition of the targeted demographic, which is not always practical.
• It overlooks the nuance that bias is often a function of the relationship between the speaker and the target, rather than just the target's identity.

The authors propose shifting the focus to Generalized Intergroup Bias, rooted in social psychology (specifically Linguistic Intergroup Bias theory). They argue that bias manifests as differences in language use based on whether the speaker and target belong to the same social group (In-group) or different groups (Out-group). The paper aims to model these interpersonal dynamics through two novel tasks:

1. Interpersonal Group Relationship (IGR) Prediction: Determining if the speaker and target belong to the same social group.
2. Perceived Interpersonal Emotion Detection: Identifying the specific emotion the speaker expresses towards the target (distinct from the overall sentiment of the tweet).

2. Methodology

Data Collection and Annotation

• Source: The authors constructed a dataset of 3,033 tweets from US Congress members (2010–2021).
• "Found Supervision" for IGR: Since political party affiliation is public metadata, the authors could automatically label tweets as In-group (same party) or Out-group (different party) without manual annotation for this specific task.
• Interpersonal Emotion Annotation: A subset of tweets was manually annotated for fine-grained interpersonal emotions using the Plutchik wheel of emotions (8 fundamental emotions: admiration, anger, disgust, fear, interest, joy, sadness, surprise).
  • Masking: To prevent annotator bias, the target's identity was masked (replaced with @Doe), and the speaker's identity was hidden.
  • Process: Three annotators per tweet selected emotions expressed by the speaker towards the target.
  • Agreement: The dataset achieved a Plutchik Emotion Agreement (PEA) score of 0.73, indicating moderate-to-high inter-annotator reliability.

Experimental Setup

The authors evaluated two tasks separately and jointly using BERTweet (a model pre-trained on 850M English tweets) as the backbone.

1. IGR Prediction Task:
   • Baselines: Sentiment-Rule (negative sentiment = out-group), Naive-Bayes SVM (NB-SVM), and BERTweet.
   • Goal: Classify if the tweet is In-group or Out-group.
2. Interpersonal Emotion Detection Task:
   • Baselines: EmoLex (a lexicon-based approach) and BERTweet.
   • Goal: Multi-label classification of the 8 Plutchik emotions.
3. Multi-Task Learning:
   • A shared encoder (BERTweet) with two output heads (one for IGR, one for Emotion) was trained simultaneously. The hypothesis was that the tasks are intertwined and could mutually reinforce each other.

3. Key Contributions

1. New Dataset: The release of the first dataset dedicated to fine-grained interpersonal emotions and interpersonal group relationships in political discourse.
2. Theoretical Shift: Moving beyond demographic-based bias detection to a relationship-based framework (IGR) that captures subtle linguistic cues of bias.
3. Empirical Findings on Human vs. Model Performance:
   • Human Limitations: Humans rely heavily on sentiment polarity (assuming positive = in-group, negative = out-group). This fails in political contexts where "bipartisan" praise (positive sentiment) is often directed at out-group members.
   • Model Superiority: Neural models significantly outperformed humans in IGR prediction, suggesting they capture latent linguistic features humans miss.
4. Multi-Task Synergy: Demonstrated that jointly modeling IGR and Interpersonal Emotion improves performance on both tasks.

4. Results

Interpersonal Group Relationship (IGR)

• Human Performance: Achieved an F1 score of 0.67 (on a small pilot), heavily reliant on sentiment heuristics.
• Model Performance:
  • Sentiment-Rule: ~56% F1 (near chance).
  • BERTweet: 74.1% F1 (significantly outperforming humans).
  • Multi-Task Model: 77.3% F1.
• Insight: The multi-task model showed a 3% improvement over the single-task BERTweet model, indicating that emotion signals help predict group relationships.

Interpersonal Emotion Detection

• Lexicon Failure: EmoLex performed poorly, highlighting that interpersonal emotions in political speech are implicit and context-dependent, not just lexical.
• Model Performance: BERTweet significantly outperformed lexicon baselines.
• Multi-Task Gains:
  • Joint modeling improved the detection of Disgust in Out-group contexts by over 10% (from 48.2% to 75.9% F1).
  • It also improved the detection of the "No Emotion" label by 20%.
  • This confirms that knowing the group relationship (IGR) helps the model distinguish subtle negative emotions like disgust, which are overwhelmingly present in Out-group interactions.

Preliminary Analysis Insights

• Emotion Distribution: Negative emotions (Anger, Disgust) are almost exclusively found in Out-group contexts. Positive emotions (Admiration, Joy) appear in both, but Admiration is more frequent in In-group contexts.
• Target Specificity: A significant portion of negative emotions (63.7% of disgust, 64.3% of anger) was directed at just three specific targets (Speakers of the House), though negative emotions were also directed at many other individuals.

5. Significance and Future Work

• Generalized Bias Modeling: This work provides a framework for studying bias that applies to any social group dynamic, not just protected demographic classes. It suggests that "debiasing" models requires understanding the speaker-target relationship.
• Linguistic Intergroup Bias (LIB): The study empirically validates LIB theory in a large-scale, real-world dataset, showing that language abstraction and emotional tone shift systematically based on group membership.
• Implications for NLP: The results suggest that standard sentiment analysis is insufficient for understanding political discourse. Models must learn to detect subtle cues (e.g., "bipartisan," "civility") that signal out-group dynamics even when the sentiment is positive.
• Limitations: The authors note that while models outperform humans, they may be exploiting dataset-specific artifacts. Future work should investigate the specific latent features models use and explore the generalizability of these findings to non-political domains.

In conclusion, the paper successfully demonstrates that interpersonal relationships are a critical, under-explored dimension of bias in NLP, and that multi-task learning combining relationship prediction and emotion detection yields superior performance in decoding the nuances of human communication.