🤖 machine learning

Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks

This paper presents the first systematic investigation into "bias inheritance," demonstrating how Large Language Models (LLMs) used for data augmentation can propagate and amplify training biases to downstream tasks, while identifying key misalignment factors and proposing three distinct mitigation strategies to address this challenge.

Original authors: Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, Jindong Wang

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, Jindong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Copycat" Problem

Imagine you are a chef trying to teach a new apprentice (a smaller AI model) how to cook. You have a few real, high-quality recipes (real data), but you don't have enough. So, you ask a famous, experienced chef (a large AI model) to write you some new recipes to fill the gap.

The problem? The famous chef has been reading millions of old newspapers and books. Some of those sources contain outdated stereotypes (e.g., "women are only good for baking," or "only men are engineers"). When the famous chef writes new recipes based on those old books, they accidentally copy those stereotypes.

This paper calls this "Bias Inheritance." It's when a new AI learns from "fake" data generated by another AI, and it accidentally inherits the prejudices of the original AI, making them even stronger.

The Experiment: Mixing the Soup

The researchers wanted to see exactly how bad this "inheritance" is. They set up a kitchen experiment:

The Ingredients: They took real, unbiased data (like a clean broth) and mixed it with "synthetic" data generated by an AI.
The Recipe: They created different types of "biased" recipes. Some were obvious (explicit), like saying "Men are better at math." Others were subtle (implicit), like using a name that implies a specific culture or gender without saying it directly.
The Taste Test: They trained new AI models on these mixed soups and then tested them on real-world tasks, like:
- Hiring: "Who should we hire for this job?"
- Salary: "How much should we pay this person?"
- Storytelling: "Write a story about a character."

What They Found: The "Echo Chamber" Effect

The results were surprising and concerning:

The Majority Gets Louder: When the AI was trained on biased data, it got better at predicting things for the "majority" group (e.g., men or Western cultures) but got worse at understanding the "minority" groups (e.g., women or non-Western cultures).
The Gap Widens: It wasn't just that the minority group did worse; the gap between the two groups got huge. For example, in salary tasks, the AI started suggesting much higher pay for men and lower pay for women, even when the resumes were identical.
The "Model Collapse" Risk: When they ran the experiment multiple times (training an AI on data made by an AI that was trained on data made by an AI), the bias got worse and worse. It was like a game of "Telephone" where the message gets distorted every time it's passed along, eventually becoming a caricature of reality.
Subtle is Dangerous: The most dangerous biases weren't the loud, obvious ones. The quiet, "implicit" biases (like using a specific name or cultural context) were actually harder to fix and caused more damage because they were hidden in the background.

Why Does This Happen? (The Three Misalignments)

The researchers found three main reasons why the AI gets confused and biased:

Value Mismatch: The AI's idea of "what people think" doesn't match what real humans actually think. If you ask an AI about cultural values, it might guess wrong because it's reading from a biased dataset, not talking to real people.
Group Mismatch: The AI just doesn't generate enough data for certain groups. If you ask it to write biographies, it might write 90 about men and only 10 about women, simply because that's what it saw in its training.
Data Mismatch: The "fake" data looks different from "real" data in the AI's brain. Even if the words are similar, the mathematical "shape" of the fake data is off, causing the AI to learn the wrong patterns.

The Solutions: Three Ways to Fix the Soup

The team tried three different methods to stop the bias from spreading, but they found that one size does not fit all.

The "Warning Label" (Token-based):
- Analogy: Putting a sticker on the recipe that says, "Warning: This recipe might be biased."
- Result: This works okay for simple tasks (like classifying a job title) because it reminds the AI to be careful. But for complex tasks (like writing a story), the AI often ignores the sticker.
The "Redaction Pen" (Mask-based):
- Analogy: Crossing out the specific words that cause bias (like "Spanish" or "Female") and replacing them with a blank space [MASK] so the AI can't see them.
- Result: This helps a little bit when the bias is very obvious. But if the bias is hidden in the way the story is told (not just the words), crossing out the words doesn't help.
The "Taste Test" (Loss-based):
- Analogy: Forcing the AI to taste its own soup and compare it to the real chef's soup. If the flavors are too different, the AI has to adjust its recipe until they match.
- Result: This was the most powerful method for fixing the "flavor" of the data, especially for complex tasks. It forces the AI to align its understanding with reality.

The Bottom Line

The paper concludes that while using AI to generate more data is a great idea, it's a double-edged sword. If we aren't careful, we aren't just copying data; we are copying and amplifying our own societal prejudices.

There is no single "magic button" to fix this. Depending on whether you are hiring, paying salaries, or writing stories, you need a different tool to stop the bias. The researchers hope this work helps future developers build systems that are fairer, rather than just faster.

Technical Summary: Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation

Problem Statement

The proliferation of Large Language Models (LLMs) has established synthetic data augmentation as a critical strategy for enhancing model performance, particularly in post-training stages where high-quality real data is scarce. However, a critical, under-explored risk exists: bias inheritance. Since LLMs inherently reflect social biases present in their pre-training corpora, using them to generate synthetic data can propagate and even amplify these biases. When downstream models are fine-tuned on such biased synthetic data, they may suffer from degraded fairness and robustness, particularly affecting minority groups. Unlike general performance collapse (e.g., model collapse due to distributional drift), bias inheritance specifically amplifies social stereotypes across demographic groups, potentially increasing unfairness even if overall accuracy metrics appear stable or improved for majority groups.

Methodology

The authors present the first systematic investigation into bias inheritance, employing a multi-dimensional framework to simulate, analyze, and mitigate the phenomenon.

1. Multi-dimensional Bias Generation Framework

To quantitatively control and study bias, the authors designed a framework generating synthetic data across six distinct bias types, derived from three primary dimensions:

Contextual vs. Contrastive: Contextual bias arises from surrounding narrative information, while contrastive bias emerges from direct comparisons between groups (e.g., selecting a candidate between two individuals with different identities).
Single vs. Intersectional: Single bias focuses on one identity attribute (e.g., gender), whereas intersectional bias considers the compounding effects of multiple attributes (e.g., gender, culture, and age).
Explicit vs. Implicit: Explicit bias involves direct references to identity, while implicit bias relies on subtle cues, such as names or cultural descriptors, that carry inherent associations.

2. Experimental Setup

Data: The study combines real, unbiased datasets (GlobalOpinionQA for cultural perspectives; BiasinBios for professional biographies) with synthetic data generated by LLMs (primarily Llama-3.1-8B-Instruct and GPT-4o-mini).
Bias Ratios ( $\gamma$ ): The proportion of biased synthetic data in the fine-tuning set is varied across $\{0\%, 5\%, 10\%, 20\%, 50\%\}$ .
Downstream Tasks: Evaluation covers 10 tasks across 17 datasets, categorized into:
- Bias Directly-Related: Tasks explicitly targeting discrimination (e.g., hate speech detection, homophobia/misogyny detection).
- Bias Indirectly-Related: Tasks where bias may influence outcomes but is not the primary target (e.g., spam detection, offensive language detection).
- Generation Tasks: Story generation and salary/hiring recommendations.
Models: Primary experiments use Llama-3.1-8B-Instruct, with cross-architecture validation on Qwen and DeepSeek series models.

3. Analysis and Mitigation Strategies

The authors analyze the root causes of bias inheritance through three "misalignment" factors and propose corresponding mitigation strategies:

Misalignment of Values: LLM responses to value-based questions diverge from human cultural responses.
- Mitigation (Token-based): Prepending tokens (e.g., "The following text may contain bias") to signal potential bias to the model.
Misalignment of Group Data: Synthetic data generation leads to imbalanced representation across demographic groups (e.g., over-representing majority groups).
- Mitigation (Mask-based): Replacing sensitive group identifiers (e.g., specific cultural labels or gendered pronouns) with placeholders like [MASK] to prevent the model from relying on biased features.
Misalignment of Data Distributions: The feature vector distribution of synthetic data differs significantly from real data in the embedding space.
- Mitigation (Loss-based): Introducing an alignment loss function during fine-tuning to minimize the Euclidean distance between the embeddings of generated text and real text.

Key Results

Understanding Bias Inheritance

Performance Disparities: Adding biased augmentation data consistently improves performance for majority groups (e.g., males, Western cultures) while degrading performance for minority groups (e.g., females, Arabic cultures).
Task Sensitivity:
- Indirectly-Related Tasks: Performance often improves at low bias ratios (10-20%), suggesting some synthetic data enhances generalization.
- Directly-Related Tasks: Performance drops significantly even with small amounts of biased data, as models learn to prioritize biased patterns over detecting inequality.
- Generation Tasks: Bias inheritance leads to wider gender pay gaps in salary recommendations and increased negative adjectives in story generation for minority cultures.
Amplification: Bias inheritance is not static; it amplifies over iterative training rounds. Eventually, the bias negatively impacts even majority groups, leading to a general performance collapse across all demographics.
Severity: Contrastive Explicit and Contextual Implicit biases exhibit the most severe negative impacts on downstream tasks.

Analysis Findings

Value Misalignment: LLMs struggle to replicate human cultural values, particularly for Eastern cultures (Arabic, Chinese), leading to significant deviations in value-based question answering.
Group Imbalance: Without explicit balancing mechanisms, LLMs generate synthetic data that reinforces existing pre-training imbalances (e.g., generating more male biographies for male-dominated professions).
Distribution Shift: Embedding analysis reveals significant distributional gaps between synthetic and real data, which correlates with performance degradation.

Mitigation Effectiveness

No single strategy universally solves the problem; effectiveness is highly nuanced:

Token-based: Effective for simple, explicit biases and classification tasks but less effective for complex cultural biases or implicit scenarios.
Mask-based: Shows promise at low bias ratios (5%) for tasks with explicit terms but fails to address subtle, distributional biases in complex generation tasks.
Loss-based: Highly effective for coarse-grained tasks and large distribution distances (e.g., salary recommendation) but struggles with fine-grained, implicit biases.

Significance and Contributions

The paper claims the following contributions to the field of LLM data augmentation:

Conceptualization: It introduces and defines bias inheritance as a distinct phenomenon from general model collapse or standard bias propagation, highlighting the specific risks of LLM-generated synthetic data.
Systematic Investigation: It provides the first comprehensive, multi-dimensional analysis of bias inheritance, covering six bias types across ten downstream tasks and multiple cultural/gender dimensions.
Analytical Framework: It identifies three key misalignment factors (values, groups, distributions) that drive bias inheritance, offering a structured way to diagnose the problem.
Mitigation Insights: It proposes and evaluates three distinct mitigation strategies, demonstrating that the choice of strategy must be tailored to the specific bias type, task complexity, and data ratio. The work underscores the substantial challenge of designing a unified solution, suggesting that future research must move beyond simple fine-tuning adjustments to address the root causes in synthetic data generation.

The authors conclude that while synthetic data is essential for the future of LLM development, the risk of bias inheritance requires rigorous, multi-faceted approaches to ensure fairness and robustness in downstream applications.