Technical Summary: Vicarious Offense and Noise Audit of Offensive Speech Classifiers

Problem Statement

Offensive speech detection is a critical component of content moderation, yet the definition of "offensive" remains highly subjective, leading to frequent disagreements between human and machine moderators. Existing literature has investigated the generalizability of detection systems across datasets, adversarial vulnerabilities, and geographic biases, but lacks comprehensive, in-the-wild evaluations of filtering outcomes on large-scale, real-world political discourse. Furthermore, current approaches often fail to account for the phenomenon of vicarious offense—the ability of an individual to predict how a person with different political affiliations will perceive a specific piece of content. This study addresses the gap in understanding the extent of disagreement among machine moderators and the limitations of human moderators in predicting cross-partisan offense perceptions.

Methodology

1. Noise Audit of Machine Moderators

The authors conducted a large-scale noise audit to measure outcome variability across different decision systems, inspired by Kahneman et al.'s concept of "noise" in human judgment.

• Dataset: A massive dataset comprising over 92 million comments from 241,112 news videos on the official YouTube channels of CNN, Fox News, and MSNBC (2014–2022). A random sample of 3 million comments was used for the audit.
• Machine Moderators (MMs): Nine distinct offensive language identification models were evaluated:
  • Eight models trained on well-known datasets (AHSD, HASOC, HatEval, HateXplain, OLID, TRAC, OHS, TCC) using BERT-LARGE-CASED architectures.
  • One publicly available model, Detoxify (ROBERTA-BASE), trained on Jigsaw's Unintended Bias dataset.
• Metric: The study quantified pairwise agreement using Cohen's $\kappa$ and overall agreement using Fleiss' $\kappa$ to determine the consistency of "offensive" vs. "not offensive" classifications.

2. Human Annotation and Vicarious Offense Study

To complement the machine audit, the authors introduced a novel annotation framework involving vicarious offense.

• Dataset Construction: A representative subset of 2,310 social web posts was created, stratified by machine moderator disagreement (easy, debated, and offensive subsets) and topic (general, gun control, reproductive rights).
• Annotators: 753 unique human annotators recruited via Amazon Mechanical Turk, balanced to ensure representation from three political identities: Democrats (35%), Republicans (35%), and Independents (30%).
• Annotation Tasks: For each post, annotators answered two types of questions:
  1. Personal (First-Person) Offense: "Do you find this offensive?"
  2. Vicarious (Second-Person) Offense: "Do you think a person with a different political identity (e.g., a Republican) would find this offensive?"
• LLM Baseline: The study also utilized the ChatGPT API (GPT-3.5-turbo) to predict vicarious offense, prompting the model to act as a specific political party member evaluating content for another party.

Key Contributions

1. Unprecedented Noise Audit: The paper presents a comprehensive noise audit revealing significant variability in content moderation outcomes across nine different machine moderators on real-world political data.
2. VOICED Dataset: The release of the VOICED (Vicarious Offense Identification Corpus for Estimating Disagreements) dataset. This is the first dataset of its kind where annotators label both first-person and vicarious offense, covering Democrats, Republicans, and Independents.
3. Vicarious Offense Framework: The introduction of a novel perspective on offense that moves beyond individual subjectivity to analyze how well political groups understand the sensitivities of opposing groups.
4. Human-Machine Alignment Analysis: A detailed analysis of the alignment (and misalignment) between human moderators of varying political leanings and machine classifiers, including the use of LLMs as vicarious predictors.

Results

Machine Moderator Disagreement

• Low Consensus: There is extensive disagreement among machine moderators. No pair of models exhibited substantial agreement ($\kappa \ge 0.81$). Only a handful showed moderate agreement ($0.41 \le \kappa \le 0.60$).
• Overall Noise: The overall Fleiss' $\kappa$ across the three news networks was low (0.27, 0.25, and 0.22), indicating that moderation outcomes vary wildly depending on the specific classifier deployed.
• Volatile Content: Nearly half of the content was flagged by no machine moderator, while a minuscule fraction (0.03%) was flagged by all. A significant portion (10.1%) fell into a "disputed zone" where exactly half the models flagged the content, making moderation outcomes highly volatile.

Human Moderator Disagreement and Vicarious Offense

• First-Person Agreement: Human-human agreement was generally higher than human-machine agreement, though still only moderate (e.g., $\kappa = 0.43$ between Independents and Democrats). Democrats and Independents were the most aligned.
• Vicarious Prediction Failure: Human moderators struggled to accurately predict how opposing political groups would perceive offense.
  • Republicans were the worst predictors of vicarious offense for both Democrats and Independents.
  • Democrats and Independents could predict each other's offense reasonably well but struggled to predict what Republicans would find offensive.
  • The "Consistency Trap": A pool of moderators consisting only of Democrats and Independents might show high consistency in predicting what Republicans find offensive, yet this prediction may be fundamentally wrong regarding what Republicans actually find offensive.

Human vs. Machine Alignment

• Political Alignment: While all political groups showed comparable agreement with machines on what is offensive, Republicans aligned slightly more with machines on what is not offensive.
• Contextual Nuance: Humans demonstrated an ability to detect offense based on context and targets (e.g., dismissive comments about Caitlyn Jenner) that machine moderators missed. Conversely, humans sometimes flagged content as offensive based on political disagreement rather than toxicity, whereas machines often relied on profanity.
• Sensitive Issues: Disagreement worsened significantly on sensitive topics like gun control and reproductive rights. On these topics, machine-human agreement dropped substantially, and some moderator pairs achieved negative Cohen's $\kappa$ values, indicating systematic disagreement.

LLM Performance

• The ChatGPT API (GPT-3.5) used as a vicarious predictor showed agreement patterns similar to the machine moderators rather than the human majority. It agreed less with human moderators and more with the machine majority, highlighting the difficulty LLMs face in modeling nuanced human political subjectivity.

Significance and Claims

The paper claims to break new ground by quantifying the "noise" in content moderation at an unprecedented scale and by introducing the concept of vicarious offense. The authors argue that:

1. Subjectivity is Fundamental: In highly polarized political discourse, there may be practically no agreement among annotator groups, leading to "alternative facts" regarding what constitutes offensive content.
2. Limitations of Current Systems: Current machine moderators and even large language models fail to predict how different political groups perceive offense, suggesting that a single "gold standard" label is often insufficient or misleading.
3. Democracy and Trust: The study highlights eroding trust in democracy, noting that different political groups hold divergent views on election fairness and the tolerance of offensive speech, which complicates the role of social media platforms in moderating discourse.

The authors position their work as primarily descriptive, aiming to "close the loop" on understanding the challenges of offensive content moderation from both machine and human perspectives, and to provide a resource (VOICED) for developing more robust, multi-perspective modeling systems. They explicitly state that their findings suggest content moderation is a nuanced topic where tolerance levels vary by issue and political identity, and that current systems are ill-equipped to handle this complexity without human oversight or more sophisticated modeling of political subjectivity.