💬 NLP

Modeling Human Perspectives with Socio-Demographic Representations

This paper introduces Socio-Contrastive Learning, a novel method that jointly models annotator perspectives and learns fine-grained socio-demographic representations to better capture the complex social contexts influencing human disagreement in NLP tasks, outperforming standard feature fusion approaches.

Original authors: Leixin Zhang, Cagri Coltekin

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leixin Zhang, Cagri Coltekin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hosting a massive dinner party where everyone is asked to judge a series of controversial jokes. Some people laugh, some are offended, and some are confused. If you just take a vote and say, "The majority said it's funny, so it's a funny joke," you miss the whole story. You lose the nuance of why the person who was offended felt that way, or why the person who laughed found it harmless.

This paper is about building a smarter computer system that doesn't just count votes, but understands who is voting and why they might see things differently.

Here is the breakdown of the research, explained with some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Mistake

In the past, AI models tried to predict if a piece of text was "hate speech" or "toxic" by looking only at the words. They acted like a judge who only reads the transcript of a trial and ignores the background of the people involved.

But humans aren't robots. Our opinions are shaped by our lives.

The Analogy: Imagine two people looking at a painting of a storm. One person, who grew up near the ocean, sees a beautiful, powerful force of nature. The other, who lost their home in a flood, sees a terrifying disaster.
The Reality: If an AI only looks at the painting (the text), it can't explain why these two people see it so differently. Previous research often tried to fix this by looking at just one trait at a time (e.g., "Do men see it differently than women?"). But the authors argue that life is more complex than a single trait. It's the mix of your age, your politics, your education, and your background all at once that shapes your view.

2. The Solution: "Socio-Contrastive Learning"

The authors created a new method called Socio-Contrastive Learning. Let's break this down into a simple metaphor.

The "Grouping" Game:
Imagine you are a teacher trying to organize a classroom. You want to seat students who have similar ways of thinking next to each other, and those who think very differently far apart.

Old Way (Concatenation): You just stick a name tag on the student's desk that says "Male, 25, College Grad." You hope the computer reads the tag and figures it out. It's a bit rigid.
The New Way (Socio-Contrastive): You don't just read the name tag. You watch how the students answer questions.
- If Student A and Student B both get the same answer on a tricky question, you move their chairs closer together.
- If Student A and Student C get opposite answers, you push their chairs far apart.
- Over time, the students naturally sort themselves into groups based on their actual perspectives, not just their name tags.

The computer does this mathematically. It looks at how people label text (e.g., "Is this toxic?") and learns to create a "mental map" of the annotators. People who agree are clustered together; people who disagree are pushed apart. This map captures the complex mix of their backgrounds automatically.

3. What Did They Find?

The researchers tested this on two big datasets: one about Hate Speech and one about Toxic Content.

Better Predictions: The new method was much better at predicting what a specific person would think compared to the old methods. It's like a weather forecaster who not only looks at the clouds but also knows the local geography and the history of the town.
The "Secret Sauce": The part of the system that did the "grouping" (the contrastive learning) was crucial. When they turned that part off, the system got worse. This proved that understanding the relationship between a person's background and their opinion is key.
The Map: When they visualized the "mental map" the computer built, they saw interesting patterns.
- People of different races tended to cluster in different areas of the map.
- People with different education levels formed their own distinct groups.
- People who considered religion very important grouped together when judging toxic content.

4. Why Does This Matter?

This isn't just about making a computer smarter; it's about making it fairer.

Avoiding the "Average" Trap: If you only train AI on the "average" opinion, you might accidentally silence minority voices. For example, if a specific group of people consistently finds a certain phrase offensive, but the majority doesn't, a standard AI might just say "It's fine."
The Goal: This new method allows us to say, "Okay, the majority thinks this is fine, but we know that for Group X, this is hurtful." It helps us build AI that respects the diversity of human experience rather than trying to flatten it into a single "correct" answer.

The Bottom Line

The authors built a tool that teaches computers to understand that context matters. Just like you wouldn't judge a book by its cover, you shouldn't judge a human's opinion without understanding their background. By using a clever "grouping" technique, they showed that when we feed an AI information about who is speaking, it becomes much better at understanding what they mean.

1. Problem Statement

In Natural Language Processing (NLP) tasks involving subjective content (e.g., hate speech and toxic content detection), human annotators often disagree. This disagreement is not merely noise but reflects valid, diverse perspectives shaped by an annotator's lived experiences and socio-demographic background.

Key Challenges Identified:

Limitations of Aggregation: Standard approaches aggregate labels (e.g., majority voting), which erases minority perspectives and creates a "ground truth" that may not exist for subjective tasks.
Simplistic Demographic Modeling: Prior work often analyzes socio-demographic factors (e.g., gender, race) in isolation or uses simple concatenation of features. This fails to capture the complex, non-linear interplay of multiple demographic attributes that shape human perspectives.
Data Scarcity per Annotator: Crowdsourced datasets typically involve thousands of annotators, each labeling only a small number of items. This makes training annotator-specific models (like multi-task heads) difficult due to insufficient data per individual.

Research Questions:

To what extent do socio-demographic features contribute to perspective prediction, and which features are most predictive?
How should socio-demographic features be encoded and fused with textual representations to maximize model performance?

2. Methodology: Socio-Contrastive Learning

The authors propose Socio-Contrastive Learning, a novel architecture designed to jointly learn annotator perspectives and socio-demographic representations.

Model Architecture

The model takes two inputs:

Text Representation: Encoded using a pre-trained transformer (e.g., Sentence-BERT).
Socio-Demographic Representation: Encoded as a multi-hot vector representing the annotator's attributes (e.g., age, race, education, political ideology).

Key Components:

Projection Layer: The multi-hot socio-demographic vector is projected into a learnable latent space.
Fusion: The projected socio-demographic vector is concatenated with the text embedding.
Prediction Head: A Multi-Layer Perceptron (MLP) predicts the specific label for that annotator.

Loss Functions

The model is optimized using a dual-objective loss function:

Cross-Entropy Loss: The primary objective to predict the annotator's specific label ( $y$ ) given the text and socio-demographic context.
Contrastive Loss: A secondary objective to refine the socio-demographic representations.
- Mechanism: For a given text instance, annotators who agree on the label are treated as positive pairs, while those who disagree are negative pairs.
- Objective: The contrastive loss pushes the socio-demographic representations of disagreeing annotators apart and pulls agreeing annotators closer together in the embedding space.
- Adaptation: Unlike standard contrastive learning (e.g., InfoNCE) which requires fixed pairs, this method handles variable numbers of annotators per text by performing contrastive learning within batches, prioritizing samples sharing the same text ID.

3. Key Contributions

Novel Architecture: Introduction of Socio-Contrastive Learning, which jointly optimizes label prediction and the learning of socio-demographic representations without requiring pre-defined demographic clusters.
Superior Performance: The method outperforms standard concatenation-based approaches (Multi-Hot Encoding) and models that ignore demographics, demonstrating that richer combinations of attributes improve prediction.
Interpretability & Analysis: The learned socio-demographic embeddings allow for the visualization and statistical analysis of how specific demographic factors correlate with perspective divergence.
Empirical Validation: Comprehensive experiments on hate speech and toxicity datasets showing that socio-demographic features are highly predictive of annotation variation.

4. Experimental Setup & Results

Datasets

Hate Speech Dataset: 6,227 unique texts, 2,315 annotators.
Toxic Content Dataset: 4,638 unique texts, 3,517 annotators.
Attributes: Education, political ideology, age, gender, race, sexuality, income, religion importance, and parental status.
Preprocessing: Converted Likert scale ratings to binary labels; filtered out annotators with too few labels.

Baselines Compared

Simple Model: Text-only, predicts aggregated (majority vote) labels.
Multi-Task Model: Text-only, predicts individual labels via separate output heads.
Socio Multi-Hot: Concatenates multi-hot demographic vectors with text embeddings.
Socio Embedding: Encodes demographics using the same pre-trained model as text, then concatenates.

Performance Results (F1 Scores)

The Socio-Contrastive Learning model achieved the best overall performance:

Hate Speech: F1 = 0.725 (vs. 0.699 for Socio Embedding and 0.608 for Multi-Task).
Toxic Content: F1 = 0.645 (vs. 0.628 for Socio Embedding and 0.543 for Multi-Task).
Ablation Study: Removing the contrastive loss resulted in a performance drop of 4.4% (Hate Speech) and 1.1% (Toxic), proving the efficacy of the contrastive mechanism in refining demographic signals.

Analysis of Socio-Demographic Contributions

Visualization (UMAP): The learned embeddings showed distinct clustering based on demographics. For example, "White" vs. "Non-white" annotators formed separate regions in the hate speech task, and education levels showed a clear progression in the toxic task.
Statistical Homophily: The authors calculated a "Ratio" (Observed Probability of nearest neighbors sharing an attribute / Expected Random Probability).
- Education showed the highest ratio (~2.6–2.8), indicating it is the strongest predictor of perspective variation.
- Political Ideology, Income, and Age also showed strong correlations (Ratio > 2).
- Gender and Race showed moderate clustering (Ratio ~1.2–1.6).
- Transgender status showed a ratio < 1, likely due to the small sample size (<5% of annotators) preventing the model from learning robust patterns.

5. Significance and Implications

Beyond Aggregation: The study validates that modeling individual perspectives is superior to majority voting for subjective tasks, capturing the nuance of human disagreement.
Feature Fusion: It demonstrates that simply concatenating demographic data is insufficient; learning a dedicated, contrastively optimized representation space yields better results.
Bias Mitigation: While the model improves overall accuracy, the authors caution that it does not automatically solve minority bias due to the small sample sizes of minority groups in test sets. However, the framework provides the necessary tools to analyze and potentially mitigate "minority-ignorance" bias in future work.
Practical Application: The method enables practitioners to understand why annotators disagree and to control the demographic composition of model outputs (e.g., simulating a specific demographic's perspective).

6. Limitations

Data Scarcity: Large-scale datasets with rich, reliable socio-demographic metadata are rare.
Attribute Constraints: The analysis is limited to the attributes provided by the source datasets; other relevant factors (e.g., specific cultural background details) were not available.
Task Generalizability: The study focuses on subjective tasks; the contribution of demographics to "objective" tasks remains unexplored.