💬 NLP

What Are They Filtering Out? An Experimental Benchmark of Filtering Strategies for Harm Reduction in Pretraining Datasets

This paper presents a benchmark study revealing that while data filtering strategies effectively reduce harmful content in pretraining datasets for Large Language Models, they inadvertently increase the underrepresentation of vulnerable groups, thereby exacerbating discrimination.

Original authors: Marco Antonio Stranisci, Christian Hardmeier

Published 2025-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marco Antonio Stranisci, Christian Hardmeier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the world's smartest librarian. This librarian (an AI) needs to read millions of books, websites, and articles to learn how to speak and think like a human. But, the internet is a messy place. It's full of great stories, but also full of hate speech, violence, and terrible stereotypes.

To make sure this librarian doesn't learn bad habits, the builders try to "filter" the books before giving them to the librarian. They want to throw away the "bad" books.

This paper is like a detective report investigating what happens when we try to clean up the library. The researchers asked two big questions:

How are people currently trying to clean the library?
Who gets accidentally kicked out of the library while we are trying to remove the bad stuff?

Here is the breakdown of their findings, using some simple analogies:

1. The "Cleaning Crew" (The Strategies)

The researchers looked at 55 different manuals from companies building these AI librarians. They found that everyone uses different cleaning tools, but they rarely tell the public exactly how they use them.

Think of the cleaning tools like different types of brooms:

The "Bad Word" Broom (Rule-based): This broom sweeps up anything with specific "dirty" words on it (like swear words or slurs).
The "Smell Test" Broom (Toxicity Classifiers): This is a robot nose that sniffs a page. If it smells "toxic," it throws the page away.
The "Quality" Broom (Document Similarity): This broom compares a page to a "perfect" page (like a Wikipedia article). If the new page doesn't look similar enough to the perfect one, it gets tossed.
The "Blacklist" Broom: This just blocks entire websites known for being trouble.

The Problem: The researchers found that most companies are being very secretive. They say, "We cleaned it!" but they won't show their work. It's like a chef saying, "This soup is safe to eat," but refusing to show you the recipe or the ingredients.

2. The Big Surprise: The "Collateral Damage"

The most important part of the paper is the experiment. The researchers took a huge pile of internet text and ran it through these different "brooms" to see what got thrown away.

They discovered a scary side effect: In trying to remove the "bad" stuff, the cleaning crew is accidentally throwing away the stories of vulnerable people, especially women.

Here is the metaphor:
Imagine you are cleaning a garden to remove weeds (harmful content).

The Goal: Remove the poisonous weeds.
The Mistake: The "Quality" broom is so aggressive that it thinks any flower that looks a little different from the "perfect" rose is a weed.
The Result: You end up with a garden full of strong, standard-looking plants (often representing men or Western perspectives), but you have accidentally pulled out almost all the rare, beautiful flowers that represent women and people from post-colonial backgrounds.

3. What Did They Find Specifically?

Women get filtered out the most: When the AI uses "Bad Word" filters or "Smell Test" robots, mentions of women disappear from the data much faster than mentions of men.
Different brooms catch different things:
- One broom catches pornography words.
- Another catches racist slurs.
- Another catches "low quality" writing.
- The lesson: If you pick one broom, you only catch one type of bad thing, but you might miss other types of bad things, or you might accidentally throw away good things that look like the bad things.
The "Quality" Trap: The researchers found that the "Quality" broom (which compares text to Wikipedia) is actually terrible at finding hate speech. It keeps a lot of toxic content but throws away a lot of unique voices. It's like judging a book by its cover and throwing away a brilliant story just because the font looks weird.

4. The "Occupation" Clue

The researchers looked at what was being thrown out.

When they filtered for "bad" content, they found that mentions of women working as actors or pornographic actors were removed at a very high rate.
Mentions of men working as politicians or writers were kept much more often.
This means the AI is learning a distorted view of the world where women are mostly seen in sexualized or removed contexts, while men are seen as leaders and creators.

The Bottom Line

The paper concludes that we are trying to fix the AI, but we are breaking the balance.

By trying to make AI "safe" by filtering out harmful content, we are inadvertently making the AI less representative of real life. We are creating a digital world where women and minorities are invisible because the "safety filters" are too blunt.

The Takeaway: We can't just use a sledgehammer to clean up the internet. We need a much more delicate, surgical approach that knows the difference between a "weed" (hate speech) and a "rare flower" (a unique perspective from a vulnerable group). If we don't fix this, our AI librarians will only know how to talk about half the world.

1. Problem Statement

The development of safe Large Language Models (LLMs) relies heavily on filtering harmful content from pretraining datasets. However, current research lacks a systematic understanding of how these filtering strategies impact vulnerable groups (e.g., women, racial minorities).

The Gap: While post-training safety measures are well-studied, the pretraining filtering stage is understudied. Existing approaches often remove harmful content but inadvertently increase the underrepresentation of vulnerable groups, creating a new form of bias.
The Question: Do current filtering strategies effectively reduce harm without disproportionately penalizing specific demographic groups?

2. Methodology

The authors employed a two-pronged approach: a systematic literature review and an experimental benchmark.

A. Systematic Survey (RQ1)

Scope: Analyzed 55 technical reports from English LMs and LLMs.
Selection Criteria: Models were selected from six leaderboards (SuperGLUE, SQuAD, MMLU-pro, Chatbot Arena, ALERT, and Secure Learning Lab) to ensure a representative mix of model generations and tasks.
Analysis: The authors extracted information on data filtering strategies, documentation, and transparency.

B. Experimental Benchmark (RQ2)

Objective: To measure the impact of specific filtering strategies on the representation of named entities categorized by gender and origin.
Demographic Groups: The study used an intersectional approach to define four groups:
1. Western Men
2. Western Women
3. Post-colonial Men
4. Post-colonial Women
Data Source: 5 samples of documents from the Common Crawl (August 2024) snapshot.
Knowledge Base: Created the "People Dataset" (10.8M entities) derived from Wikidata and CaLiGraph to link named entities to demographics (birth country, citizenship, gender, ethnicity).
Pipeline:
1. Entity Linking: Used SpaCy for NER, Wikipedia API for disambiguation, and Wikidata for ID matching.
2. Filtering Implementation: Applied 7 specific filtering strategies to the dataset:
  - Rule-based: Shutterstack Lexicon, HateBase.
  - Classifier-based: Perspective API, FastText (trained on Jigsaw), Profanity-check.
  - Quality-based (Document Similarity): Classifiers trained on Wikipedia vs. OpenWebText to filter low-quality content.
3. Metric: Calculated the percentage of named entity mentions removed by each strategy compared to the unfiltered baseline.

3. Key Contributions

First Systematic Survey: Provided a taxonomy of 8 distinct filtering strategies found in literature (Authoritative Sources, Document Seeding, Quality-based, Toxicity Classifiers, Rule-based, URL Blacklists, Human-in-the-loop, Safety Policy).
Benchmark Pipeline: Developed a reproducible pipeline and resource (People Dataset) to evaluate filtering strategies against demographic underrepresentation.
Empirical Evidence: Demonstrated that harm reduction strategies often have the side effect of increasing the underrepresentation of vulnerable groups, specifically women.

4. Key Results

A. Transparency and Replicability

Low Transparency: Most modern LLMs (e.g., GPT-4, Llama-3) do not disclose their filtering scripts or dataset snapshots.
Documentation Debt: While early models (BERT) were open, and a "second generation" (C4) was partially open, recent reports often provide only generic descriptions (e.g., "we filter for safety") without technical details.

B. Impact of Filtering Strategies (Quantitative)

Magnitude of Removal:
- Rule-based/Classifier-based: Marginal impact (removing <1% to ~4% of sentences).
- Quality-based (Document Similarity): Aggressive impact (removing 11% to 44% of content).
Demographic Disparity:
- Women are the most impacted: In rule-based and classifier-based strategies, mentions of Western Women are removed at the highest rates (e.g., -4.2% with Shutterstack vs. -2.3% for Western Men).
- Quality-based Paradox: While quality-based filters remove the most content overall (impacting men slightly more), the authors found these filters do not effectively remove toxic content. They remove safe content while leaving harmful content intact.

C. Nature of Filtered Content

Lexicon Bias: Different lexicons target different harms.
- Shutterstack: Focuses on pornography (e.g., "sex," "nude").
- HateBase: Focuses on racism and misogyny (e.g., "slaves," "dykes").
Occupational Bias: Filtering disproportionately removes specific professions for women.
- For Western women, the most removed occupation was "pornographic actor" (not a top occupation in the general dataset).
- For Post-colonial women, "model" was heavily filtered.
- This suggests filters are removing specific types of representation rather than just "harm," leading to the erasure of specific female identities.

5. Significance and Conclusion

Trade-off Identified: There is a direct trade-off between reducing harmful content and preserving demographic representation. Current strategies prioritize removing specific types of "toxicity" (often sexualized or racial slurs) but fail to do so without disproportionately silencing women and post-colonial voices.
Quality $\neq$ Safety: Relying on "high-quality" sources (like Wikipedia) as a proxy for safety is flawed; these sources still contain significant harmful content and do not guarantee the removal of bias.
Call to Action: The paper argues for:
1. Finer-grained analysis of filtering impacts beyond simple toxicity scores.
2. Participatory evaluation involving communities affected by discrimination.
3. Transparency in publishing filtering scripts and datasets to allow for independent auditing.

Limitations: The study relies on named entities, potentially missing mentions via pronouns or demonyms. Additionally, the underlying knowledge bases (Wikidata/Wikipedia) have their own biases (underrepresentation of non-Western and female figures), which may skew the baseline data.