💬 NLP

Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

This paper introduces Perspective-Driven Inference, an adaptive sampling method that leverages a limited human annotation budget to accurately estimate the distribution of LLM annotations across diverse demographic groups in subjective tasks, thereby addressing the limitations of single ground-truth assumptions.

Original authors: Navya Mehrotra, Adam Visokay, Kristina Gligorić

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Navya Mehrotra, Adam Visokay, Kristina Gligorić

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how different groups of people feel about a new movie. You ask 1,000 people for their reviews, but you only have the budget to pay 20 of them to write detailed, honest reviews. The other 980 reviews are generated by a very smart, but slightly biased, robot.

The problem? The robot is great at guessing how a 20-year-old tech enthusiast feels, but it's terrible at guessing how a 70-year-old retiree feels. If you just let the robot guess for everyone, your final report will be wrong about the retirees. If you ask the 20 humans to review random movies, you might accidentally ask 19 of them to review movies the robot already got right, and miss the retirees entirely.

This paper introduces a clever new strategy called Perspective-Driven Inference (PDI) to solve this exact problem. Here is how it works, using simple analogies:

1. The Problem: The Robot's Blind Spots

Large Language Models (LLMs) are like super-fast, super-cheap robots that can read text and give opinions (like "Is this email polite?" or "Is this comment offensive?").

The Catch: These robots have "blind spots." They often understand the perspective of a young, educated person perfectly, but they might completely misunderstand an older person or a person from a different cultural background.
The Old Way: Researchers used to either trust the robot blindly (which leads to bad data) or ask humans to review a random mix of texts (which is inefficient and might miss the groups the robot is bad at).

2. The Solution: The "Smart Scout" Strategy

The authors propose a method where you use a tiny bit of human effort to fix the robot's biggest mistakes. Think of it like a Scout Leader managing a team of Robot Scouts.

Step 1: The Robot Does the Heavy Lifting. The robot reads every single text and gives a quick rating. It's fast and cheap, but we know it's sometimes wrong.
Step 2: The "Burn-in" (The Warm-up). The system asks a few humans to rate a random handful of texts. This gives the system a tiny taste of reality to compare against the robot's guesses.
Step 3: The Smart Scout (Adaptive Loop). This is the magic part. The system looks at the robot's guesses and the few human answers it already has. It asks: "Where is the robot looking confused?"
- If the robot is confident but the humans disagree (especially for a specific group, like people over 50), the system flags that text as "High Priority."
- If the robot is already accurate for a group, the system says, "No need to waste human money here."
Step 4: Targeted Human Help. Instead of asking humans to review random texts, the system directs the human reviewers specifically to the texts where the robot is most likely to be wrong for specific groups.

3. The Result: A Fairer Picture

By spending the human budget only where it's needed most, the researchers get a much more accurate picture of how different groups feel.

Without this method: You might get a report that says "Everyone thinks this is polite," because the robot guessed that for everyone.
With this method: You get a report that says, "Young people think this is polite, but older people find it rude," because you specifically asked humans to check the texts where the robot failed to understand the older perspective.

Why This Matters (The "So What?")

In the real world, we often use AI to analyze social science data, like detecting hate speech or measuring customer satisfaction.

If we don't fix the robot's bias, we might think a policy is working well because the robot says so, while actually, it's hurting a specific group of people that the robot doesn't understand.
This method ensures that no one's voice is lost just because the AI is bad at simulating them. It treats the "disagreement" between groups not as a mistake to be fixed, but as a valuable signal to be preserved.

The Takeaway

Think of this paper as a budget-saving guide for fairness. It teaches us how to use a little bit of expensive human wisdom to correct a cheap, fast AI, ensuring that the final result represents everyone, not just the groups the AI happens to be good at understanding. It's about knowing where your robot is blind and sending a human guide to that specific spot.

1. Problem Statement

Large Language Models (LLMs) are increasingly used to annotate texts in computational social science to reduce human labor. However, standard LLM annotation methods face two critical limitations in subjective tasks (e.g., rating politeness or offensiveness):

Absence of Ground Truth: In subjective domains, "correctness" is not a single scalar value but a distribution of perspectives. Reasonable people from different demographic groups often disagree on whether text is toxic, polite, or empathetic. Treating this disagreement as "error" to be corrected by a majority vote discards meaningful signal.
Demographic Bias in LLMs: LLMs do not represent all demographic perspectives equally. They often default to a specific worldview (e.g., Western, educated, young) and fail to capture the nuances of underrepresented groups (e.g., older adults, specific racial minorities).
Failure of Existing Corrections: Current "repair" methods (like Prediction-Powered Inference) assume a single ground truth exists. Furthermore, techniques like persona prompting (asking the LLM to "act as" a specific demographic) have been shown to be unreliable, often exaggerating stereotypes or failing to improve accuracy for the very groups they aim to represent.

The Core Challenge: How can researchers use limited human annotation budgets to estimate the distribution of annotations across demographic groups (a vector estimand) rather than a single average, while correcting for systematic LLM bias?

2. Methodology: Perspective-Driven Inference (PDI)

The authors propose PERSPECTIVE-DRIVEN INFERENCE (PDI), a framework that treats the distribution of group-specific annotations as the quantity of interest. The method combines LLM proxies with an adaptive sampling strategy to allocate human annotation effort efficiently.

Key Components:

Vector Estimand: Instead of estimating a single scalar $\theta^*$ , the goal is to estimate a vector $\theta^* = (\theta^*_{g_1}, \dots, \theta^*_{g_K})$ , where each entry represents the mean annotation for a specific demographic group $g_k$ .
Adaptive Sampling Loop:
1. Burn-in Phase: Collect a small initial set of human annotations uniformly at random to establish a baseline.
2. Error Prediction: Train a black-box predictor (e.g., XGBoost) to estimate the squared LLM error $(\hat{H}_i - H_i)^2$ based on the annotator's demographic features ( $d_i$ ).
3. Adaptive Allocation: Calculate the probability of sampling a new human annotation for text $i$ as proportional to the predicted error: $\pi_i \propto \widehat{\text{err}}_i(d_i)$ . This directs the human budget toward groups where the LLM is predicted to be least accurate.
4. Iteration: Update the error predictor with new data and repeat the sampling process.
Estimation & Correction: Once the budget is exhausted, the framework uses Inverse Probability Weighting (IPW) to produce unbiased group-level estimates and valid confidence intervals. This corrects for the non-uniform sampling distribution.

3. Key Contributions

Formal Framework: The paper formalizes multi-perspective corpus inference as the estimation of a vector estimand, extending standard PPI (Prediction-Powered Inference) methods to settings where no single ground truth exists.
Adaptive Sampling Strategy: A novel method that concentrates human annotation resources on demographic groups where LLM proxies are least reliable, rather than using uniform sampling.
Empirical Evaluation: Comprehensive evaluation on politeness and offensiveness rating tasks using real-world data (POPQUORN) and synthetic data, demonstrating that PDI significantly outperforms baselines for hard-to-model groups.
Critique of Persona Prompting: The study provides strong evidence that persona prompting is an unreliable substitute for actual human input from specific demographic groups, often failing to capture the intended perspectives.

4. Experimental Results

The authors evaluated PDI against several baselines: LLM-only (zero-shot, few-shot, persona), and PPI with uniform sampling.

Datasets: Politeness and offensiveness ratings from POPQUORN (demographically diverse human annotations) and a synthetic binary dataset.
Models: Tested across 8 LLMs (including GPT-5.2, Claude, Gemini, Llama).
Key Findings:
- Coverage: LLM-only methods frequently failed to achieve valid coverage (often <50% for specific groups like Age 50+). PDI maintained coverage above 90% across all groups.
- Accuracy (Delta): PDI significantly reduced the Mean Absolute Error (MAE) for underrepresented groups.
  - Example (Offensiveness, Age 50+): LLM-only few-shot had a delta of 24.54%, while PDI achieved 5.24%.
  - Example (Politeness, Age 50+): PDI reduced delta from 16.31% (few-shot) to 11.23%.
- Efficiency: The adaptive strategy successfully identified high-error groups. For the Age 50+ group in the politeness task, PDI allocated 33% more human annotations than uniform sampling, directly translating to improved accuracy.
- Persona Prompting Failure: Persona prompting often resulted in the worst performance, sometimes yielding 0% coverage for specific groups, confirming it cannot replace human data.

5. Significance and Implications

Valid Inference for Subjective Tasks: PDI provides a statistically rigorous way to analyze subjective data without collapsing diverse human perspectives into a single "average" that may not represent anyone.
Resource Efficiency: By adaptively sampling, researchers can achieve high-fidelity results for minority or hard-to-model groups with the same budget that would yield poor results under uniform sampling.
Broader Applications: The framework is applicable beyond simple rating tasks to:
- RLHF: Addressing the bias in preference data where models are trained on a narrow slice of human feedback.
- LLM-as-a-Judge: Correcting for the fact that a single model cannot fairly judge outputs across all demographic contexts.
- Survey Simulation: Improving the accuracy of simulating public opinion distributions.
Ethical Considerations: The authors caution that while PDI improves statistical validity, it concentrates annotation labor on marginalized groups. They emphasize the need for fair compensation and informed consent for these participants.

Conclusion

The paper demonstrates that correcting LLM annotation bias requires more than just better prompting; it requires a statistical framework that acknowledges the existence of multiple valid perspectives. Perspective-Driven Inference offers a robust solution by adaptively allocating human effort to where LLMs fail, ensuring that downstream analyses faithfully represent the diversity of human opinion rather than the biases of the model.