Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation

Technical Summary: Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation

Problem Statement
Human label variation is a central phenomenon in Natural Language Processing (NLP), particularly in subjective tasks like harmful language detection. Recent shifts in data collection practices have moved toward increasing annotator numbers and releasing disaggregated datasets to capture individual labeling decisions rather than aggregating them into a single "gold" label. While this provides rich data regarding who annotated (sociodemographics, attitudes), the what (linguistic properties of the items) and the interplay between the two have received limited attention. Existing research often fails to account for the cross-classified structure of annotation data (where items and annotators are non-nested factors) and frequently ignores interactions between annotator characteristics and item features. This paper addresses the gap in understanding "who differs in their perception of what" by conducting a large-scale, principled analysis of annotation variation across four reference datasets.

Methodology
The authors analyze four English harmful language datasets (CTDP, MHS, POPQUORN, D3CODE), comprising over 25,000 items, 8,000 annotators, and 205,000 annotations. The study employs a Bayesian multilevel regression framework with the following key components:

1. Data Structure: The models account for the partially cross-classified data structure by including random intercepts for both annotators and items. This addresses the systematic variation present at both levels.
2. Feature Engineering:
   • Annotator Characteristics: Up to 19 features per dataset, including sociodemographics (age, gender, race, education, etc.) and attitudes (moral foundations, political ideology, item-specific attitudes).
   • Linguistic Features: Over 300 features extracted using the elfen package, covering surface-level, readability, psycholinguistic, part-of-speech, lexical richness, morphological, information-theoretic, dependency, semantic, named entity, and emotion/sentiment features. Additionally, domain-specific lexical signals (e.g., slurs, vulgarities) were extracted from multiple hate speech lexicons.
   • Interactions: The models explicitly test interactions between annotator features, between linguistic features, and crucially, between annotator and item features.
3. Modeling Approach: Bayesian multilevel regression with Horseshoe priors is used to handle the high dimensionality of predictors (up to 5,264 fixed effects) and multicollinearity. This regularization approach aggressively shrinks small and uncertain effects toward zero, allowing the identification of "surviving" effects (those with 90% posterior credible intervals not overlapping zero).
4. Experimental Scenarios: The analysis is conducted in three realistic scenarios:
   • Comparing effects across different datasets with related tasks but different conceptualizations.
   • Simulating new annotators by splitting the D3CODE dataset to assess intra-group variation.
   • Simulating new batches by splitting the CTDP dataset to assess cross-batch stability.

Key Results
The analysis reveals that annotation variation is driven by complex interactions rather than simple main effects of annotator demographics or item features alone:

• Dominance of Interactions: Interactions between annotator characteristics and linguistic features are crucial. For instance, in the POPQUORN dataset, the effect of hateful words on offensiveness ratings varies significantly by age, with older annotators showing a stronger correlation. In MHS, ideological stance interacts with age to produce stark differences in ratings, particularly at the extremes of the political spectrum.
• Linguistic Cues: Lexical cues remain strong predictors. Features such as negative sentiment, the presence of hateful/vulgar tokens, and specific named entities (e.g., nationalities, religious groups) consistently influence annotations. However, the specific cues vary by dataset and task.
• Attitudes vs. Demographics: In the D3CODE simulation, sociodemographic proxies (like age or gender) often failed to survive as main effects, whereas moral foundations (specifically "care") and item-specific attitudes (e.g., whether an annotator thinks a post should be removed) were significant.
• Intersectionality: The study identifies intersectional effects that are often ignored in prior work. For example, the interaction between ideology and age in MHS revealed that conservative older annotators rated content differently than liberal older annotators, a pattern not visible when looking at age or ideology in isolation.
• Dataset Specificity: Effect patterns vary considerably across datasets. While negative sentiment was a common survivor, other effects were highly specific to the dataset's annotation guidelines, item composition, and annotator pool. This suggests limited generalizability of findings from one specific configuration to another.
• Spurious Signals: The analysis uncovered potential data artifacts, such as an interaction where Hindu annotators rated items with non-standard tokens (X tags) as less hateful, likely due to sampling coincidences rather than a genuine linguistic effect.

Significance and Contributions
The paper makes two primary contributions:

1. Methodological: It establishes a principled approach for analyzing large-scale disaggregated datasets, demonstrating the necessity of using multilevel models with random intercepts and regularization to handle the cross-classified structure and high-dimensional feature spaces. It highlights the importance of modeling interactions to disentangle variation.
2. Substantive: It provides the first large-scale assessment of annotation behavior in harmful language detection from a linguistic, annotator-item interaction, and intersectional perspective.

The authors argue that answering "who annotates what" is vital for multiple NLP communities. For the harmful language detection community, it offers insights into effect patterns that can inform better data collection strategies (e.g., ensuring diverse annotator representation for specific item types). For the human label variation community, it underscores the need to consider both the "who" and the "what" to understand variation and guide targeted re-annotation. Finally, for the modeling and content moderation communities, the findings warn against treating label variation as mere noise; instead, systematic differences in how specific groups perceive specific text items should be preserved to avoid flattening nuanced perceptions and to identify potential spurious confounders in model training.

The paper concludes with a call for caution regarding the generalization of findings across different datasets and tasks, emphasizing that annotation is an interactive process between complex human identities and specific linguistic compositions.