From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

1. Problem Statement

The paper addresses a fundamental challenge in Natural Language Processing (NLP): capturing subjective human perspectives, particularly those of specific demographic or cultural subgroups (e.g., toxicity detection, safety judgments).

• The Core Issue: Unlike factual tasks, subjective judgments lack an objective ground truth. They depend on lived experience, social context, and individual bias.
• Current Limitation: Traditional annotation pipelines rely on aggregating human judgments. However, recruiting "in-group" annotators (e.g., non-binary individuals to judge non-binary perspectives) is often costly, slow, or impossible for small or intersectional groups.
• The Prevailing Assumption: Large Language Models (LLMs) are currently treated as a pragmatic fallback—a cost-saving compromise used only when human annotation is unavailable, but inherently inferior to "real" human perspectives.
• Research Question: Can LLMs actually outperform humans in estimating aggregate group perspectives, and under what specific conditions?

2. Methodology: Perspective-Taking as Statistical Estimation

The authors reframe Perspective-Taking (PT) not as a measurement of individual opinion, but as the statistical estimation of a latent group-level mean.

Theoretical Framework

The paper formalizes the problem where the target quantity $f^*(x, g)$ is the expected judgment of a random member of group $g$ on item $x$. Both humans and LLMs are viewed as estimators of this latent mean. The authors decompose the Mean Squared Error (MSE) of an estimator into three structural components:
$$MSE = \text{Bias}^2 + \text{Correlation Floor} + \text{Reducible Variance}$$

They introduce a "Two-Lens" model to analyze bias:

1. Wide Lens (Representation Bias): How well the estimator's internal knowledge reflects the target population's distribution.
   • Humans: Limited by lived experience and social circles (sampling bias).
   • LLMs: Limited by pretraining data coverage and potential stereotypes.
2. Clear Lens (Processing Bias): How the internal representation is translated into a numeric judgment.
   • Humans: Subject to cognitive distortions and projection.
   • LLMs: Subject to alignment constraints and safety fine-tuning.

Key Hypothesis: Coupling and Super-Additivity

• Humans: Social identity often creates a positive coupling between representation and processing errors (e.g., an out-group annotator both misrepresents the group and projects their own norms in the same direction). This leads to super-additive bias ($\mu^2_{total} > \mu^2_{repr} + \mu^2_{proc}$).
• LLMs: Representation (pretraining) and processing (post-training/inference) are mechanically distinct stages. This decoupling often results in lower variance and weaker coupling, potentially reducing total error.

Experimental Setup

• Datasets:
  • Toxicity Detection: 120 comments with dense annotations from US-based crowdworkers (including a new non-binary cohort). Ground truth is the mean of direct in-group annotations.
  • DICES: A conversational safety dataset with rich demographic axes (race, age, education) to test boundary conditions.
• Protocols:
  • Direct Annotation: In-group humans rate items for themselves.
  • Perspective-Taking (PT): Annotators (human or LLM) estimate the percentage of a target group that would find an item toxic/unsafe.
• Evaluation: Mean Squared Error (MSE), Bias, and Variance across varying annotation budgets ($k=1$ to $10$).

3. Key Contributions

1. Formalization of PT as Estimation: Shifts the comparison from "authenticity" to "estimation quality," enabling a principled bias-variance-correlation comparison between humans and LLMs.
2. Bias-Variance-Correlation Framework: Introduces a decomposition that explains why humans fail in out-group PT (super-additive coupling) and why LLMs succeed in low-budget regimes (low variance, decoupled errors).
3. The "Reasoning Paradox": Identifies that explicit Chain-of-Thought (CoT) reasoning can degrade LLM performance in PT tasks by inducing "criterion drift" (shifting from estimating empirical group rates to applying rigid rubric-based classifications).
4. Boundary Conditions: Defines clear regimes where LLMs are superior vs. where human judgment remains essential.
5. Differential Perspective-Taking (DPT): A diagnostic tool to isolate group-specific sensitivity from generic annotation skill, revealing where LLMs fail to capture nuanced group differences.

4. Key Results

A. The Budget Regime Hypothesis (H1)

• Low Budget ($k=1$): A single LLM consistently outperforms a single human (both in-group and out-group) in predicting aggregate judgments.
  • Reason: Humans have high variance ($V_H$) due to individual differences. LLMs are relatively deterministic ($V_L \ll V_H$) under fixed prompting.
  • Finding: A single LLM PT estimate is often statistically equivalent to aggregating 3–5 direct human labels.
• High Budget: As $k$ increases, human performance improves but plateaus due to a "correlation floor" (shared cultural norms). LLM ensembles show modest gains because they are already low-variance.

B. The Coupling Hypothesis (H2)

• Out-Group Human PT: Suffers from massive error increases due to super-additive coupling. When an out-group human estimates another group's view, representation errors and processing errors align, inflating total bias.
• LLM Stability: LLMs, lacking social identity, do not suffer from this identity-mediated coupling. Their error remains stable across different target groups, making them robust when in-group recruitment is infeasible.

C. Representation Limits (H3)

• Specificity & Prevalence: LLM performance degrades as groups become highly specific (e.g., "Black, college-educated women") or low-prevalence (e.g., non-binary, specific racial minorities).
• Mechanism: Sparse or stereotype-skewed training data leads to large representation bias ($b_{repr}$). In these regimes, human in-group knowledge remains superior.

D. The Reasoning Paradox

• Finding: Enabling native reasoning modes (CoT) often increases bias and MSE.
• Mechanism: Reasoning causes criterion drift. The model shifts from estimating "what fraction of this group thinks X" to "is X objectively toxic according to my internal rubric?" This re-couples representation and processing, reintroducing the super-additive error humans suffer from.

E. Engineering Insights

• Prompting: Structured prompts (definitions, levels) can shift bias but do not universally reduce error.
• Model Mixing: Combining predictions from different model families yields modest error reductions by canceling out bias profiles.
• Temperature: Increasing sampling temperature provides limited gains, as the primary error source is often bias, not variance.

5. Significance and Implications

Repositioning LLMs:
The paper argues that LLMs should not be viewed merely as a "fallback" for missing human data. Instead, they are statistically superior frontline estimators in specific, well-defined regimes:

1. Low-budget scenarios where human variance dominates.
2. Broad or heterogeneous groups where in-group humans still suffer from representation errors.
3. Out-group estimation where human social identity induces super-additive bias.

When Humans are Essential:
Human annotation remains indispensable for:

1. Deep, specific, and cohesive subgroups (e.g., intersectional minorities) where LLM training data is sparse.
2. Differential sensitivity: Tasks requiring the detection of subtle per-item differences between groups (DPT analysis showed humans outperform LLMs here).
3. Procedural Legitimacy: High-stakes contexts where stakeholder participation and lived experience are ethically required, regardless of statistical optimality.

Conclusion:
The future of perspective-taking lies in principled integration. LLMs should handle aggregate estimation in variance-dominated regimes to free up scarce human expertise for cases requiring contextual nuance, legitimacy, and deep subgroup sensitivity. The paper provides a theoretical and empirical roadmap for engineering these hybrid pipelines.