💬 NLP

IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Language

This paper demonstrates that automated content moderation tools fail to capture the heterogeneous and subjective attitudes of marginalized communities toward reclaimed slurs, leading to the misclassification of solidarity-driven language as hate speech due to a lack of contextual understanding and significant disagreement even among in-group members.

Original authors: Christina Chance, Rebecca Pattichis, Arjun Subramonian, James He, Shruti Narayanan, Saadia Gabriel, Kai-Wei Chang

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christina Chance, Rebecca Pattichis, Arjun Subramonian, James He, Shruti Narayanan, Saadia Gabriel, Kai-Wei Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "One-Size-Fits-All" Robot vs. The Complex Human Heart

Imagine you are trying to teach a robot how to understand human language. You give the robot a list of "bad words" (slurs) and tell it: "If you see this word, delete the post immediately because it's hate speech."

The robot does exactly what you ask. It deletes everything containing those words.

But here is the problem: The robot doesn't understand context, history, or friendship.

For many marginalized communities (like Black people, LGBTQIA+ people, and women), these same "bad words" have been taken back and used as a way to show love, solidarity, and pride among friends. It's like a group of friends using a nickname that would be an insult if a stranger used it, but a term of endearment if a best friend uses it.

This paper argues that current AI moderation tools are like that clumsy robot. They are too dumb to tell the difference between a friend joking and a stranger attacking. Because they can't tell the difference, they accidentally silence the very people they are supposed to protect.

The Core Experiment: Asking the Community to Judge

The researchers didn't just guess; they asked members of these communities to act as judges. They gathered about 12,000 tweets containing reclaimed slurs (the "n-word," the "f-word," and the "b-word") and asked community members to label them.

They asked questions like:

"Is this person using the word to be mean, or to show pride?"
"If the author is part of our community, should we report this?"
"If the author is not part of our community, should we report this?"

The Shocking Discovery: "We Don't All Agree"

The researchers expected that if they asked members of the same community, they would all agree. They thought, "If we ask Black people about the n-word, they will all say the same thing."

They were wrong.

The Analogy: Imagine a potluck dinner where everyone brings a dish. You expect everyone to agree on the recipe. Instead, you find that some people think the dish is a delicious family tradition, others think it's a funny inside joke, and some think it's actually gross.

The study found massive disagreement even among people from the same community.

One person might see a tweet and think, "That's a proud reclaiming of our identity!"
Another person from the same community might see the exact same tweet and think, "That's still hurtful and should be banned."

Why? Because everyone has a different life story.

Maybe one person grew up hearing the word used only as a weapon, so they can't stand to see it used anywhere.
Maybe another person grew up hearing it used as a joke among cousins, so they feel it's safe to use.

The paper calls this "Heterogeneous Attitudes." It means: There is no single "correct" way to feel about these words, even within the same group.

The Robot vs. The Humans

The researchers also tested a popular AI tool called Perspective API (which many social media sites use) against the human judges.

The AI's Logic: "I see the word. It is a slur. Therefore, it is hate speech. Delete it."
The Humans' Logic: "I see the word. Who said it? To whom? In what context? Is it a joke? Is it a quote? Is it a new slang meaning?"

The Result: The AI was often wrong.

It frequently flagged friendly, reclaimed usage as hate speech (False Positives).
It failed to understand that the same word might be acceptable if a Black person says it, but unacceptable if a white person says it.
The AI seemed to assume that everyone using the word was an outsider trying to be mean.

The "B-Word" vs. The "N-Word" Analogy

The paper found that different words are treated differently, even by the same people.

The "B-Word" (B*tch): This word has become so common in movies, music, and marketing that it's almost like a regular noun. The AI and humans agreed more on this one. It's like a word that has lost most of its sharp edge.
The "N-Word": This word carries a heavy history of violence and trauma. The disagreement here was huge. The AI was very aggressive in flagging it, often hurting Black users who were using it among themselves. The humans were much more nuanced, understanding that context is everything.

The Takeaway: We Need a "Human-in-the-Loop"

The paper concludes that we cannot rely on a simple "Gold Standard" (a single set of rules) for moderation.

The Metaphor:
Trying to moderate reclaimed language with a simple AI is like trying to judge a complex family argument by only looking at the volume of the voices. You miss the tone, the history, and the relationship.

What should we do?

Stop assuming everyone in a community thinks alike. We need to accept that people will disagree, and that disagreement is normal.
Context is King. We need systems that can read the room, not just the words.
Don't let the robot be the only judge. We need human nuance, especially from the communities being affected, to decide what is actually harmful.

In short: The paper says, "Hey, AI, you're doing a bad job at understanding the complex, messy, beautiful, and painful ways humans use language to connect. You're hurting the people you're trying to help. We need to do better."

1. Problem Statement

Automated content moderation systems and AI-based hate speech detectors frequently fail to distinguish between hateful usage and reclaimed usage of slurs (e.g., the n-word, f-word, b-word) within marginalized communities.

The Core Issue: Current systems rely on keyword matching or static toxicity models that lack the contextual nuance to understand reclamation. This leads to the over-moderation and suppression of marginalized voices.
The Assumption Gap: A prevailing assumption in NLP research is that "in-group" annotators (members of the marginalized community) provide a homogeneous, reliable "gold standard" for labeling reclaimed language. This paper challenges that assumption, hypothesizing that intra-community disagreement is substantial and that lived experience creates heterogeneous interpretations that cannot be captured by a single consensus label.
The Consequence: Automated tools (like Perspective API) are calibrated to dominant cultural norms, often misinterpreting reclaimed language as hate speech, thereby reinforcing digital colonialism and silencing community expression.

2. Methodology

The study employs a mixed-methods approach combining quantitative annotation analysis with qualitative interviews.

A. Communities and Slurs

The research focuses on three marginalized groups and their specific reclaimed slurs:

Black Community: n-word
LGBTQIA+ Community: f-word
Women Community: b-word (b*tch)

B. Data Collection

Corpus: Approximately 12,000 tweets containing the target slurs were collected via the X (Twitter) API and existing hate speech datasets (Dynahate, ToxiGen, Jigsaw).
Selection: 100 texts per slur (300 total) were selected for annotation, balancing toxic and non-toxic classifications based on Detoxify.
Annotators: 21 in-group annotators (6 for n-word, 9 for b-word, 6 for f-word) recruited from specialized AI communities (Black in AI, Queer in AI, Women in ML).
Interviews: 7 semi-structured interviews were conducted with a subset of annotators to explore the role of lived experience.

C. Annotation Protocol

Annotators labeled each text with the following features:

Reclamation Type: Pride reclamation (expression of dignity) vs. Insular reclamation (in-group camaraderie) vs. Neither.
Derogation: Is the usage derogatory? (Binary Yes/No).
Target: Who is the slur directed at? (Self, in-group, out-group, etc.).
Context: Salient context categories (e.g., counter-speech, quote, neologism, discussion of identity).
Reporting Decision (Counterfactual): Would you flag this as hate speech if the author were in-group vs. out-group?

D. Analysis Techniques

Inter-Annotator Agreement: Calculated using Krippendorff's Alpha ( $\alpha$ ) to measure consensus.
Feature Importance: An XGBoost model was trained to identify which text features (e.g., derogation, target, context) most strongly predict the decision to flag content as hate speech.
Model Alignment: Human judgments were compared against Perspective API's "Identity Attack" scores using Average Total Variation (ATV) to measure calibration gaps.
Qualitative Analysis: Thematic analysis of interview transcripts to understand the drivers of disagreement.

3. Key Results

A. Low Intra-Community Agreement (RQ1)

Contrary to the assumption that in-group members agree on reclamation, agreement scores were extremely low:

Krippendorff's $\alpha$ for reclamation type (Q1) was near zero or negative for all slurs (n-word $\alpha = 0.05$ , f-word $\alpha = 0.07$ , b-word $\alpha = 0.18$ ).
Even for the "hate speech reporting" question, agreement was low (n-word $\alpha = 0.17$ ).
Implication: Disagreement is not "noise" but a signal of the subjective, context-dependent nature of reclaimed language. A single "gold standard" label is impossible to establish.

B. Predictive Features of Reporting (RQ2)

Author Identity: The assumed group membership of the author was the most frequent feature used by the model to predict reporting decisions.
Contextual Nuance: Whether a text was derogatory and whether the target was oneself were strong predictors.
Slur-Specific Logic: Different slurs activated different interpretive heuristics. For example, the n-word in "discussion of slur" contexts heavily influenced reporting based on authorship, whereas the f-word showed different patterns.

C. Poor Model Calibration (RQ3)

Perspective API Bias: The API's identity attack scores were significantly misaligned with human judgments, particularly for in-group usage.
Average Total Variation (ATV):
- n-word: High ATV for in-group (0.42) vs. low for out-group (0.14). The model assumes n-word usage is an attack unless the author is out-group, failing to recognize in-group reclamation.
- b-word: Lowest in-group ATV (0.18), suggesting the model is slightly better calibrated for this term, likely due to its broader mainstream normalization.
- f-word: High ATV for both groups, indicating poor alignment.
Conclusion: Automated models disproportionately suppress in-group reclaimed speech, validating user experiences of self-censorship.

D. Heterogeneous Interpretations (RQ4)

Cultural & Generational Differences: Interviews revealed that age, geography (e.g., US vs. Nigeria), and specific community sub-groups (e.g., trans vs. cis gay men) drastically alter the acceptability of a slur.
Ambiguity: Terms like the b-word have shifted from purely misogynistic to flippant/camaraderie uses, creating confusion even among in-group members.
Context Collapse: Without metadata (profile info, network), text-only analysis fails to capture the "who" and "why," leading to misinterpretation.

4. Key Contributions

Empirical Evidence of Heterogeneity: Demonstrates that "in-group" is not a monolith; shared identity does not guarantee consensus on hate speech or reclamation.
Critique of Gold Standards: Challenges the NLP community's reliance on aggregated in-group labels as the ground truth for training moderation models.
Quantification of Algorithmic Harm: Provides concrete metrics (ATV) showing how current state-of-the-art models (Perspective API) systematically fail marginalized communities, particularly regarding the n-word.
Methodological Framework: Introduces a counterfactual annotation approach (asking annotators to judge based on hypothetical author identity) to isolate the impact of authorship on moderation decisions.

5. Significance and Implications

For NLP Research: The paper argues for a shift from consensus-based annotation to pluralistic approaches. Models should be designed to handle heterogeneous judgments rather than forcing a single label.
For Content Moderation: Current "one-size-fits-all" policies are insufficient. The authors suggest Personalized Content Moderation (PCM) or Pluralistic Alignment, where systems acknowledge multiple valid interpretations of the same text based on community norms.
Ethical Considerations: The study highlights the risk of "digital colonialism," where automated systems enforce Western, dominant-culture norms on marginalized groups, suppressing their cultural expression and solidarity.
Future Directions: The authors advocate for context-sensitive approaches that incorporate user identity and community-specific norms, rather than relying solely on text-based keyword detection.

Conclusion: The paper concludes that automated moderation cannot currently capture the complex, subjective, and heterogeneous attitudes of marginalized communities toward reclaimed language. Effective moderation requires moving beyond simple classification toward systems that respect the plurality of human interpretation and the critical role of context.