💬 NLP

Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails

This paper audits pretraining filters and inference-time guardrails in language models, revealing that they disproportionately erase mentions of marginalized groups through reliance on simplistic lexical cues while failing to catch actual hate speech or private information, thereby creating a form of epistemic injustice that contradicts human judgment.

Original authors: Marco Antonio Stranisci, A Pranav, Rossana Damiano, Christian Hardmeier, Anne Lauscher

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marco Antonio Stranisci, A Pranav, Rossana Damiano, Christian Hardmeier, Anne Lauscher

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive library of books to teach a super-smart robot how to speak and understand the world. This robot, called a Large Language Model (LLM), needs to read billions of sentences from the internet to learn.

However, before the robot starts reading, humans put up filters (like a strict librarian) to throw out "bad" books. Then, when the robot starts talking to people, humans put up guardrails (like a safety inspector) to stop it from saying anything "unsafe."

This paper is an audit—a deep investigation—into how these librarians and safety inspectors are doing their jobs. The researchers found that while they are trying to keep the robot safe, they are accidentally erasing the voices of marginalized groups (like transgender people, women, and people from Central America) and replacing human judgment with rigid, often mistaken, rules.

Here is a breakdown of their findings using simple analogies:

1. The "Banned Word" Trap

The researchers found that many of these safety systems work like a child's "naughty list."

How it works: If a sentence contains a specific "bad" word (like a swear word or a sexual term), the system immediately deletes it, without looking at the context.
The Problem: It's like a librarian throwing out a book about a medical procedure just because it mentions a body part, or deleting a story about a sex worker's rights because it uses the word "sex."
The Result: The systems are very good at catching words on their "naughty list," but they are terrible at spotting actual harm, like privacy leaks, copyright theft, or real hate speech that doesn't use the specific banned words.

2. The "Over-Protective" Filter

The study discovered that these filters are over-protective against specific groups, acting like a bouncer who is suspicious of everyone wearing a certain type of hat.

Transgender People: Sentences mentioning transgender identities were flagged for removal much more often than sentences about cisgender (non-trans) people.
Women: Mentions of women were flagged significantly more often than mentions of men.
Central Americans: Content mentioning people from Central America was almost always flagged for removal, while content about people from Western Europe was rarely touched.
The Analogy: Imagine a security guard at a museum who lets everyone in except for people from one specific neighborhood. The guard isn't trying to be mean; they are just following a flawed rule that assumes people from that neighborhood are "risky." This results in the museum (the AI) never learning about that neighborhood's culture.

3. The Robot vs. The Human

The researchers asked human volunteers to look at the sentences the robots flagged and decide: "Should this stay or go?"

The Shocking Stat: The humans said "Keep it" 88.5% of the time for sentences the pre-training filters wanted to delete, and 91.3% of the time for sentences the guardrails wanted to block.
The Analogy: It's like a robot chef tasting a soup, deciding it's "poisonous" because it contains a specific herb, and throwing the whole pot away. A human chef tastes it, realizes it's just a spicy soup, and says, "No, this is delicious and safe to eat."
The Conflict: The automated systems are removing content that humans actually find valuable, educational, or harmless. By deleting these sentences, the AI is losing the ability to understand these groups and their experiences.

4. The "Silent Erasure"

The paper calls this "Epistemic Erasure."

What it means: "Epistemic" relates to knowledge. "Erasure" means wiping something out.
The Metaphor: Imagine a map of the world where the AI is drawing the continents. Because the filters keep deleting sentences about transgender people or Central Americans, the AI's map ends up with blank white spots where those groups should be. The AI literally doesn't "know" they exist or what their lives are like because the data was scrubbed away before it could learn.

Summary of the Paper's Claims

The Systems Don't Agree: The different filters and guardrails often disagree with each other. Some catch things others miss, and they rely heavily on simple word lists rather than understanding context.
Bias is Systemic: This isn't just a glitch; it's a pattern. Systems from different companies and different continents all seem to over-flag the same marginalized groups.
Human Judgment is Different: Humans generally see these sentences as safe or important, while the machines see them as dangerous.
The Consequence: By relying on these automated systems, we are actively shaping AI to be blind to the realities of transgender people, women, and people from specific regions, effectively silencing them in the digital world.

The paper concludes that to fix this, we can't just rely on automated "naughty lists." We need to involve the people being affected in the decision-making process and create systems that understand context rather than just counting bad words.

Technical Summary: Epistemic Injustice in Language Models

Problem Statement

Modern Large Language Models (LLMs) rely on two primary mechanisms to mitigate harm: pretraining filters to remove undesirable content from training corpora and inference-time guardrails to suppress unsafe outputs during deployment. While these systems are standard in AI pipelines, their impact on marginalized social groups remains poorly understood. Existing research has largely focused on bias in hate speech detection or benchmark performance, leaving a gap in understanding the broader representational effects of filtering and moderation. Specifically, it is unclear whether these systems align with one another, whether they align with human judgment, and whether they systematically erase the identities and perspectives of marginalized communities, thereby producing epistemic injustice.

Methodology

The authors conducted an audit of four pretraining filters and three inference-time guardrails using a sample of Common Crawl sentences. The study focused on identity mentions related to gender and regional origin, categories central to bias literature and extractable via curated lexicons and Named Entity Recognition (NER).

Data and Systems

Corpus: A filtered subset of Common Crawl (English, >5 sentences, >10 words/sentence) yielding over 100,000 documents.
Identity Extraction: Entities were extracted using NER and linked to Wikidata. Gender was categorized into five groups (man, woman, non-binary, questioning, genderfluid) with a trans/cis modality axis. Regional origin was mapped to 17 UN Macro-Regions.
Filters Audited:
- HATEBASE: Rule-based lexicon for racial and misogynist slurs.
- SHUTTERSTOCK: Rule-based sexual vocabulary blocklist (used in C4/T5).
- DOLMA: FastText classifier trained on Jigsaw toxicity data.
- HATEBERT: BERT re-trained on Reddit posts from banned communities.
Guardrails Audited:
- Llama-Guard-3-8B: Fine-tuned on Meta's usage policy (14 risk categories).
- MD-Judge-v0.1: LLM-based judge fine-tuned on SALAD-Bench (6 domains, 66 categories).
- Qwen3Guard-Gen-8B: Generative variant trained on a multilingual harm taxonomy including a "controversial" label.
Human Annotation: A stratified subset of 500 sentences was manually annotated by two linguistics students. The annotation task framed harm detection as a trade-off between the cost of retaining vs. removing content, capturing justifications for both keeping and removing flagged text.

Analytical Framework

The study employed two analytical splits:

Flagged Set: Sentences flagged by at least one filter (72,659 sentences) for inter-system alignment and identity-conditional flag-rate analysis.
Unflagged Set: Sentences from the same source documents where no filter fired (257,210 sentences) for guardrail-only analysis.
Statistical significance was assessed using Cohen's $\kappa$ for agreement and Wilson confidence intervals for flag rates, with Benjamini–Hochberg correction for multiple comparisons.

Key Findings

1. Disagreement Between Filters and Guardrails

Lexicon Dependence: Guardrails showed strong correlation with the SHUTTERSTOCK blocklist (sexual vocabulary), but weak correlation with other filters (HATEBASE, DOLMA, HATEBERT).
Missed Harm Categories: Pretraining filters largely failed to detect categories that guardrails frequently flag, specifically Intellectual Property, Privacy, and Political Content. For instance, 68.6% of Llama-Guard's unsafe verdicts on unflagged sentences concerned IP and Privacy, categories no pretraining filter is designed to detect.
Guardrail Divergence: While MD-Judge and Qwen3Guard showed substantial agreement ( $\kappa \approx 0.60$ ), Llama-Guard acted as an outlier, showing only moderate agreement with the others, likely due to its reliance on Meta's specific usage policy rather than a broader academic harm taxonomy.

2. Epistemic Erasure of Marginalized Identities

The audit revealed systematic over-flagging of specific identity groups across all systems, despite their training on different taxonomies and continents:

Transgender Identities: Mentions of transgender people were flagged significantly more often than cisgender mentions (e.g., SHUTTERSTOCK flag rates rose from 48.5% to 76.8%).
Women: Mentions of women were flagged significantly more often than men across all guardrails and most filters.
Central Americans: This group experienced the highest deviation, with Llama-Guard, Qwen3Guard, and SHUTTERSTOCK flagging nearly 96–99% of mentions, compared to a baseline of ~26%.
Western Regions: Conversely, mentions of Western European, Western Asian, and Northern European origins were significantly under-flagged.

3. Tensions in Content Moderation (Human vs. Machine)

Human annotators disagreed with automated systems in the vast majority of cases:

Retention Rates: Annotators would retain 88.5% of content flagged by filters and 91.3% of content flagged by guardrails.
Representational Harms: The study identified specific tensions where removal causes harm:
- Sex Work: Removing content often strips occupational visibility for sex workers, while keeping it can perpetuate sexualized commodification.
- Politics: Removing flagged political viewpoints leads to a loss of diverse political representation in model outputs.
- Health/Practical Info: Over-flagging medical "how-to" information (e.g., wart examination) risks degrading model capabilities in health domains.
- Racialized Objectification: Systems often fail to distinguish between harmful racialized objectification and legitimate group representation.

Significance and Claims

The paper claims to document a form of epistemic erasure where mentions of marginalized groups are disproportionately removed before pretraining and suppressed again at inference time. The authors argue that these systems do not merely reduce harmful content but actively shape whose identities, perspectives, and experiences remain visible in modern language models.

The findings suggest that current filtering and moderation approaches are insufficient for pluralistic alignment. The authors propose that achieving epistemic justice requires:

Transparency: Releasing filter lexicons with documentation on flag rates across identity groups.
Reflexivity: Involving affected communities in defining harm taxonomies and annotator pools rather than imposing a "view from nowhere."
Data Sovereignty: Addressing biases at the source of data collection and ownership.

The study concludes that human evaluators systematically disagree with automated systems, viewing the removal of much flagged content as unnecessary and a source of bias against specific categories of people. The authors position this work as a call to move beyond technical harm reduction toward socio-technical solutions that preserve the epistemic diversity of marginalized communities.