💬 NLP

The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data

This paper identifies and demonstrates "fairness collapse," a phenomenon where language models trained on synthetic data exhibit amplified social biases and degrade in fairness metrics significantly earlier than they show performance declines in standard language modeling tasks.

Original authors: Irina Proskurina, Antoine Gourru, Julien Velcin

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Irina Proskurina, Antoine Gourru, Julien Velcin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write stories by feeding it a massive library of human books. This is how modern "Large Language Models" (LLMs) learn: they read trillions of words to understand how humans speak, think, and describe the world. But here is the catch: the internet is getting flooded with stories written by robots themselves. Soon, a robot might be reading stories written by another robot, which were written by a robot before that, creating a giant loop of machine-generated text.

Scientists have already discovered a scary problem called "model collapse." If a robot only reads robot-written stories, it starts to forget the real world. Its vocabulary shrinks, its sentences get repetitive, and it eventually stops making sense, like a photocopy of a photocopy that gets blurrier and blurrier with every copy. But there is a second, sneakier danger. We know these robots sometimes learn human stereotypes—like thinking "nurse" always means a woman and "engineer" always means a man. The big question was: if a robot keeps training on its own biased, robot-written stories, does it just get worse at writing, or does it get more biased? This paper dives into that specific corner of computer science to see if the robot's prejudice gets worse before its writing skills actually break.

The researchers set up a controlled experiment to watch this happen in real-time. They took a language model and made it write thousands of fake professional biographies (like short stories about doctors, nurses, and engineers). Then, they fed those fake stories back into the model to teach it again, repeating this cycle over and over. They did this in two ways: one where the robot learned from fresh human data mixed with its own fake stories, and another where it only learned from its own previous fake stories, creating a closed loop.

The team found something surprising and unsettling. They discovered a phenomenon they call "fairness collapse." Usually, when a robot starts to fail, we see obvious signs: its writing becomes gibberish, or it gets terrible at answering simple questions. But in this study, the robot's writing actually improved at first. Its confusion scores (called perplexity) went down, meaning it was getting better at predicting the next word in a sentence. However, while its writing looked perfect, its internal bias was getting worse and worse.

Think of it like a student who is studying for a test by only reading notes written by a friend who is terrible at math. The student might start memorizing the friend's notes perfectly, getting higher and higher scores on practice quizzes because the notes are consistent. But the student is actually learning the wrong math. In the same way, the robot in the study started to "memorize" its own biased stories. As it kept training on its own output, the link between "nurse" and "female" (or "engineer" and "male") became stronger and stronger, even though the robot's general writing skills were still looking great.

The study showed that this bias amplification happened before the robot started making obvious mistakes. In one of their experiments, after five rounds of training on its own fake data, the robot's bias score jumped significantly, while its general knowledge test scores only started to drop slowly. This suggests that the robot is becoming more prejudiced in a "silent" way. It's not crashing; it's just becoming more convinced of its own stereotypes.

The researchers also found that this "fairness collapse" was much worse when the robot trained exclusively on its own previous outputs (the recursive loop) compared to when it had some fresh human data mixed in. In the closed loop, the robot's bias grew steadily and predictably, reinforcing the same stereotypes over and over.

So, what does this mean? It suggests that as we rely more on AI to generate content for the internet, we might be creating a feedback loop where AI models become increasingly biased without us noticing. The "warning lights" that usually tell us a model is failing (like bad writing or low test scores) might not turn on until it's already deeply biased. The paper concludes that we need new ways to check for fairness, because by the time a model looks like it's "breaking," it might have already become a very confident, very prejudiced version of itself.

Technical Summary: The Fairness Collapse Phenomenon

Problem Statement

As machine-generated text increasingly contaminates the training corpora of language models (LLMs), there is a critical risk of "model collapse," a phenomenon where recursive training on synthetic data leads to a progressive loss of distributional diversity and performance degradation. While prior work has established that model collapse manifests as a decline in standard language-modeling metrics (e.g., perplexity) and downstream task performance, it remains unclear whether this process amplifies or attenuates existing social biases.

The authors hypothesize a distinct failure mode termed "fairness collapse." This phenomenon posits that recursive training on self-generated data creates a feedback loop where demographic stereotypes (specifically gender-occupation associations) are silently amplified across generations. Crucially, the authors propose that this bias amplification may occur before traditional indicators of model collapse (such as perplexity increases or accuracy drops) become apparent, posing a significant risk where models degrade in fairness while appearing to improve or remain stable in standard metrics.

Methodology

Data and Model

Dataset: The study utilizes the Bias in Bios corpus, a collection of human-written professional biographies annotated with gender and occupation. To isolate the effects of recursive training from inherent dataset imbalances, the authors constructed a balanced subset containing 27,752 biographies across 28 professions (approx. 2M tokens).
Base Model: Experiments were conducted using Qwen2.5-0.5B.
Training Regimes: The authors compared three controlled continued-pretraining strategies (illustrated in Figure 2):
1. Human-only: Repeated training on the original human corpus (baseline).
2. Iterative Regeneration: Synthetic data is generated by the model from the previous iteration ( $M_{\theta_t}$ ), but the next training run initializes from a fresh base model ( $M_{\theta_0}$ ). This isolates the effect of regenerated data from cumulative parameter updates.
3. Recursive Contamination: The model is continuously trained on its own generated outputs. $M_{\theta_t}$ generates synthetic data and serves as the initialization for the next training iteration ( $M_{\theta_{t+1}$ ), creating a closed feedback loop.

Synthetic Data Generation

Two generation strategies were employed to create synthetic biographies:

Seeded Generation: Models continue human-written prefixes (seeds) of varying lengths (5 to 50 tokens). This allows precise control over the proportion of model-generated content within a sample.
Few-Shot Generation: Models generate biographies based on prompts containing $K=3$ human-written examples of a specific profession.

Evaluation Protocol

Fairness Metrics:
- Occupation Prediction Accuracy: A pairwise classification task determining if a biography matches its true profession versus a semantically related alternative.
- Equality of Opportunity (EO) GAP: Measures disparities in true positive rates between male and female biographies for each profession. The aggregate score is the root mean square (RMS) of profession-level gaps.
- Likelihood Asymmetry: Signed conditional likelihood gaps (NLL-GAP) measuring token-level probability differences between genders within the same profession.
- External Benchmarks: CrowS-Pairs (stereotype preference) and SoFA (social fairness bias).
General Performance Metrics:
- Perplexity (PPL): Measured on the training data used at each iteration.
- MMLU: Multitask accuracy across diverse domains (science, law, medicine, etc.) to monitor general capability degradation.
Data Quality Metrics: Wasserstein distance (embedding distribution alignment) and MTLD (lexical diversity).

Key Results

1. Fairness Degradation Precedes Model Collapse

The central finding is that fairness degradation emerges significantly earlier than conventional indicators of model collapse.

In recursive seeded training, the EO GAP increased from 13.18 (Iteration 0) to 19.38 (Iteration 5).
Concurrently, perplexity improved (decreased) from 16.07 to 10.55, and MMLU accuracy declined only gradually (from 42.14 to 32.08).
Similar trends were observed in few-shot generation, where fairness metrics deteriorated despite stable or improving language-modeling performance in early iterations.

2. Amplification of Demographic Associations

Recursive training reinforced existing biases rather than introducing arbitrary noise:

Directional Drift: Female-associated professions (e.g., "nurse") shifted toward increasingly negative EO values (indicating stronger bias against females in those roles), while male-associated professions (e.g., "professor") drifted toward positive values.
Likelihood Asymmetry: Metrics like NLL-GAP and CrowS-Pairs showed progressively stronger demographic asymmetries across iterations, even as the model's ability to predict the next token (perplexity) improved.

3. Recursive vs. Iterative Regimes

Recursive Contamination produced the most severe and monotonic fairness degradation. The compounding effect of updating the same model on its own outputs accelerated bias amplification.
Iterative Regeneration (re-initializing the base model each time) exhibited slower and less consistent fairness degradation, though bias amplification was still present. This suggests that while exposure to biased synthetic data is the primary driver, the recursive parameter update loop exacerbates the effect.

4. Decoupling of Lexical Diversity and Fairness

Analysis of lexical diversity (MTLD) and Wasserstein distances revealed that fairness shifts can accumulate even in the absence of dramatic surface-level distributional collapse. While MTLD scores remained relatively stable, the group-conditioned disparities (EO GAP) moved significantly, indicating that aggregate metrics may fail to capture the erosion of fairness.

Significance and Claims

The paper claims to identify a distinct early-stage failure mode in language model training: Fairness Collapse.

Silent Amplification: The primary significance is the demonstration that bias can increase "silently" while standard metrics (perplexity, MMLU) suggest the model is performing well or even improving. This challenges the assumption that monitoring standard language-modeling metrics is sufficient to detect degradation in model safety or fairness.
Risk of Synthetic Contamination: As synthetic text becomes a larger proportion of web corpora, the recursive feedback loop described here poses a critical risk. Models trained on such data may become increasingly biased against specific demographic groups before any obvious collapse in generation quality is detected.
Limitations: The authors modestly note that their findings are specific to the Bias in Bios dataset (gender/occupation in English), relatively small decoder-only models, and fixed generation configurations. They do not claim these results generalize universally to all model scales or sensitive attributes without further validation.

In conclusion, the work provides empirical evidence that the "curse of recursion" extends beyond distributional collapse to include the silent amplification of social biases, necessitating new monitoring strategies that specifically track fairness metrics alongside traditional performance indicators.