💬 NLP

From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP

This position paper argues that Human Label Variation (HLV), often dismissed as noise, should be preserved as an intrinsic value (*Selbstzweck*) in post-training NLP datasets to better capture human pluralism and ensure sociotechnical safety, rather than collapsing diverse perspectives into artificial consensus.

Original authors: Shanshan Xu, Santosh T. Y. S. S, Barbara Plank

Published 2026-04-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shanshan Xu, Santosh T. Y. S. S, Barbara Plank

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Stop Erasing the "Messy" Human Voice

Imagine you are trying to teach a robot how to be a good friend. You ask 100 different people to rate the robot's answers.

Person A says, "That answer was too short!"
Person B says, "That answer was too long!"
Person C says, "I liked the tone, but the facts were wrong."

In the old days of AI, researchers would look at this mess and say, "This is noise! We need to get rid of the disagreement." They would force the 100 people to vote, pick the winner, and throw away the 99 other opinions. They treated human disagreement like a typo or a mistake that needed to be fixed.

This paper argues that we need to stop doing that.

The authors say that human disagreement isn't a mistake; it's a feature. It's proof that humans are diverse, complex, and have different values. Instead of trying to flatten all those differences into one "perfect" answer, we should treat that variety as the most important thing we have.

They call this shift a journey through three stages:

Stage 1: The "Noise" Era (Human Adjacent)

The Metaphor: The Factory Line
Imagine a factory making identical widgets. If a worker puts a widget on the line that looks slightly different, the manager yells, "That's a defect! Throw it in the trash!"

What happened: Early AI was like this factory. Researchers thought there was only one "Gold Standard" truth for every question. If two people disagreed, one of them was "wrong."
The Problem: This ignored the fact that humans aren't widgets. We have different backgrounds, cultures, and feelings. By throwing away the "defects," the AI learned to be rigid and boring, missing the richness of human life.

Stage 2: The "Signal" Era (Human Aware)

The Metaphor: The Weather Report
Later, scientists realized that the "defects" weren't random errors; they were actually data.

What happened: Researchers started saying, "Wait, if 30% of people hate this joke and 70% love it, that's not an error. That's a signal telling us the joke is subjective."
The Shift: They started treating disagreement as useful information to make AI more robust. But, they still mostly tried to average it out to get a single score.

Stage 3: The "Selbstzweck" Era (Human Centered)

The Metaphor: The Symphony vs. The Solo
Selbstzweck is a German word meaning "an end in itself." The authors argue that human diversity shouldn't just be a tool to make AI better; it should be the goal itself.

The Big Idea: Imagine a choir. In the old days, we tried to make everyone sing the exact same note (a solo). In the "Signal" era, we realized different notes make a chord. But in this new era, we realize the harmony itself is the masterpiece.
The Goal: We shouldn't force AI to pick a single "winner" opinion. Instead, the AI should learn to understand, respect, and navigate the fact that people will always disagree. The AI's job isn't to find the "truth," but to navigate the plurality of human values.

Why Does This Matter Now? (The "Post-Training" Problem)

We are currently in the era of Large Language Models (LLMs) like the one you are talking to. To make these models helpful and safe, we use a process called RLHF (Reinforcement Learning from Human Feedback).

The Current Flaw:
Right now, when we train these models, we take thousands of human opinions and crush them into a single "Yes/No" or "1 to 5 star" score.

Analogy: Imagine you are a judge in a court case. Instead of reading the arguments of the jury, the judge just takes a vote, picks the majority, and throws away the dissenting opinions. The final verdict is "The Majority Wins."
The Risk: This creates an AI that thinks there is only one way to be "good." It ignores minority voices, cultural nuances, and complex ethical dilemmas where there is no single right answer.

What Should We Do Instead? (The Solutions)

The authors propose three simple changes to fix this:

Hire a Better "Jury" (Annotator Pool):
Don't just hire 100 people from the same university or country. Hire a diverse group that represents the real world (different ages, cultures, backgrounds). If your jury is diverse, their disagreement is valuable data, not a mistake.
Show the Whole Scorecard (Release Individual Labels):
Instead of telling the AI "The average score is 4 stars," show the AI the raw data: "Person A gave 5 stars, Person B gave 2 stars, Person C gave 4 stars." Let the AI learn that people have different tastes, rather than forcing it to learn a fake "average" taste.
Listen to the "Why" (Diverse Feedback):
Don't just ask people to pick "Option A" or "Option B." Ask them to explain why they chose it. Sometimes, people might say, "I don't prefer either; they are both good in different ways." The AI needs to understand these "ties" and nuances, not just force a winner.

The Legal Analogy: The Dissenting Opinion

The paper uses a great legal example. In a Supreme Court, sometimes the judges can't agree.

Old Way: The court issues a ruling based on the majority vote and ignores the minority.
New Way (The Paper's Suggestion): The court issues the majority ruling (because a decision must be made), BUT they also publish the "Dissenting Opinion." This preserves the minority view for history, accountability, and future reasoning.
For AI: Our AI shouldn't just give one answer. It should be able to say, "Most people think X is better, but a significant group thinks Y is better because of Z. Here is the context for both."

Summary

This paper is a call to action for the AI community: Stop trying to clean up human disagreement.

Human disagreement is not "noise" to be deleted. It is the signal of our diversity. And ultimately, preserving that diversity is the goal (the Selbstzweck) of building AI that truly serves all of humanity, not just a simplified, averaged version of it.

Based on the position paper "From Noise to Signal to Selbstzweck - Reframing Human Label Variation in the Era of Post-training in NLP" by Xu, Santosh, and Plank, here is a detailed technical summary.

1. Problem Statement

The paper addresses a critical flaw in the current paradigm of Natural Language Processing (NLP) and Large Language Model (LLM) alignment: the systematic erasure of Human Label Variation (HLV).

The Core Issue: Traditionally, NLP annotation pipelines treat inter-annotator disagreement as "noise" or error to be eliminated, often aggregating multiple annotations into a single "gold" label via majority vote.
The Consequence: In the era of post-training (e.g., Reinforcement Learning from Human Feedback - RLHF), this practice collapses diverse human perspectives into an artificial consensus. This flattens the pluralism of human values, leading to models that are aligned with a narrow, averaged view rather than the full spectrum of human needs and societal contexts.
The Gap: While recent work acknowledges HLV as a signal for robustness, current preference datasets (e.g., standard RLHF datasets) rarely preserve annotator-level variation. Furthermore, the conceptual understanding of HLV has not evolved to treat it as an intrinsic value (Selbstzweck) essential for sociotechnical safety.

2. Methodology and Conceptual Framework

The paper does not present new empirical experiments but rather proposes a conceptual framework and prescriptive strategies based on a synthesis of existing literature and sociotechnical theory.

A. The Three-Stage Evolutionary Framework

The authors propose a trajectory for how HLV and human pluralism are conceptualized in NLP, paralleling the shift from "Human Adjacent" to "Human Centered" systems:

Human Adjacent (Noise): Early NLP (pre-1990s). Disagreement is viewed as error. Systems are optimized for narrow, objective tasks (e.g., POS tagging) by experts. Evaluation focuses on accuracy/F1 against a single "truth."
Human Aware (Signal): The rise of crowdsourcing and deep learning (1990s–2022). Disagreement is recognized as a signal of task ambiguity and human subjectivity (e.g., hate speech detection). Evaluation begins to consider human interaction and bias, but datasets still often aggregate labels.
Human Centered (Selbstzweck): The LLM era (Post-2023). HLV is elevated to Selbstzweck (a German term meaning "end in itself" or intrinsic value). Pluralism is not just a signal to be modeled but a guiding design principle. The goal is to build systems that reflect the diversity of human values, not just a single consensus.

B. Sociotechnical Evaluation Integration

The framework integrates the Weidinger et al. (2023) sociotechnical safety evaluation model, arguing that alignment must move beyond:

Layer 1 (Capability): Technical performance in isolation.
Layer 2 (Human Interaction): Contextual safety.
Layer 3 (Systemic Impact): Broader societal effects (e.g., polarization).
Preserving HLV is essential for evaluating models at Layers 2 and 3.

3. Key Contributions

The paper offers three primary contributions:

Conceptual Reframing: It argues that HLV must be treated as Selbstzweck—an intrinsic value representing human pluralism—rather than a nuisance to be minimized.
Critical Analysis of Preference Datasets: It identifies that current datasets (e.g., Multi-Pref, HelpSteer2, Ouyang et al.) often aggregate annotations, failing to capture systematic, legitimate divergence in human preferences. It highlights that aggregation can actively harm pluralistic alignment by rewarding models for "decisive" choices even in low-agreement scenarios.
Actionable Strategies for Dataset Construction: The authors propose four specific interventions to embed HLV into future alignment pipelines:
- Careful Annotator Pool Selection: Move beyond opaque recruitment. Use survey methodology (stratified quotas, multi-channel recruitment) to ensure the annotator pool reflects the diversity of the target population, mitigating selection bias.
- Release Annotator-Level Preferences: Instead of publishing only mean scores or aggregated labels, datasets should release individual annotator labels. This enables distributional reward models that capture uncertainty and heterogeneity, rather than collapsing them into a single scalar.
- Leverage Diverse Feedback Types: Move beyond simple pairwise comparisons (which favor assertive responses). Incorporate heterogeneous feedback protocols, including scalar ratings, $n$ -wise comparisons, and descriptive evaluations, to capture fine-grained perspectives.
- Pluralism as Reasoning: In scenarios requiring a single decision (e.g., legal rulings), systems should preserve pluralism in the reasoning process (documenting dissenting opinions) even if the final output is unified, similar to legal court records.

4. Results and Evidence

As a position paper, the "results" are derived from the synthesis of existing empirical studies cited in the text:

Prevalence of HLV: Citing Zhang et al. (2025), the paper notes that over 30% of examples in major preference datasets (Multi-Pref, HelpSteer2) contain disagreement, with fewer than 25% attributable to pure annotation error.
Impact of Aggregation: Evidence suggests that aggregating labels during preference training forces models to make decisive choices in cases of genuine disagreement, thereby undermining pluralistic alignment.
Legal and Social Analogues: The paper draws parallels to legal systems (recording dissenting opinions) and AI mediation tools, demonstrating that preserving disagreement is feasible and beneficial for accountability.

5. Significance and Future Directions

The paper is significant because it challenges the foundational assumption of "Gold Standard" labels in the post-training era.

Paradigm Shift: It calls for a shift from viewing disagreement as a data quality issue to viewing it as a feature of human values.
Safety and Ethics: By preserving HLV, models can be better evaluated for sociotechnical safety, ensuring they do not inadvertently suppress minority viewpoints or enforce a false consensus on sensitive social issues.
Future Research Directions:
- New Metrics: The authors propose metrics like Pluralism Fidelity (measuring how well a model's output distribution matches the empirical human response distribution) and Minority Regret (quantifying utility loss for minority preference groups).
- Algorithmic Development: Future alignment algorithms should move beyond scalarization to distributional modeling of disagreement, enabling conditional personalization and continual learning of evolving norms.
- Scope Expansion: While focused on post-training data, the authors note that the Selbstzweck principle should eventually extend to pretraining corpus construction and filtering.

In conclusion, the paper argues that for NLP to achieve genuinely human-centered AI, the community must stop treating human variation as noise and start treating it as the core objective of alignment.