💬 NLP

How Does Differential Privacy Affect Social Bias in LLMs? A Systematic Evaluation

This paper systematically evaluates the impact of differential privacy on social bias in large language models across four paradigms, revealing that while DP reduces bias in sentence scoring tasks, it does not uniformly improve fairness across all tasks and that reduced memorization does not necessarily equate to reduced unfairness.

Original authors: Eduardo Tenorio, Karuna Bhaila, Xintao Wu

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eduardo Tenorio, Karuna Bhaila, Xintao Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a very smart robot to write stories and answer questions. You feed it a massive library of books from the internet. The problem is, the internet is full of old stereotypes (like "nurses are women" or "wealthy people are smart"). If the robot memorizes these books too perfectly, it might start repeating these stereotypes, which can be unfair.

To stop the robot from memorizing specific, sensitive details, scientists use a technique called Differential Privacy (DP). Think of DP as a "fog machine" for the robot's learning process. It adds a little bit of static noise to the lessons, making it harder for the robot to remember any single specific fact. This protects privacy.

But here is the big question the paper asks: Does this "fog" also make the robot less biased? Does blurring the memories of specific people also blur the robot's unfair stereotypes?

The researchers tested this using a robot named VaultGemma-1B (the only one currently available that was trained with this privacy fog) and compared it to two standard robots trained without the fog. They didn't just ask one question; they used four different "games" to see how the robots behaved.

Here is what they found, broken down by game:

1. The "Fill-in-the-Blank" Game (Sentence Scoring)

The Setup: The robot is given a sentence with a blank, like "The manager said that ___ was not very educated." It has to choose between "the poor person" (a stereotype) or "the wealthy person" (not a stereotype).
The Result: The robot trained with the privacy fog (DP) did much better here. It was less likely to pick the stereotypical answer.
The Analogy: Imagine the robot is a student taking a multiple-choice quiz. The privacy fog made the student less confident in their "gut feelings" based on stereotypes. Instead of blindly guessing the cliché, they hesitated and chose more balanced options. In this specific game, privacy helped reduce bias.

2. The "Story Writing" Game (Text Completion)

The Setup: You give the robot a prompt like "The woman is described as..." and ask it to finish the sentence.
The Result: The results were messy. Sometimes the privacy robot was fairer, sometimes it wasn't. It depended on whether you were measuring "politeness," "toxicity," or "sentiment."
The Analogy: This is like asking the student to write a short story instead of picking an answer. Even if the student didn't memorize the stereotypes (thanks to the fog), they might still write them down because of how they are talking or the tone they use. The "fog" didn't automatically fix the story; it just made the robot's writing a bit more unpredictable. Privacy didn't guarantee fair stories.

3. The "Data Table" Game (Tabular Classification)

The Setup: The robot is shown a list of facts (Age, Job, Race) and asked to predict if someone earns over $50k.
The Result: The robots were terrible at this game, regardless of whether they had privacy fog or not. They guessed randomly.
The Analogy: The researchers realized they were asking the wrong question. They asked the robot to act like a data analyst, but the robot was trained to be a storyteller. It was like asking a poet to do long division. Because the robot was so confused by the format, the privacy fog didn't matter at all. The game itself was broken, so we couldn't tell if privacy helped.

4. The "Multiple Choice" Game (Question Answering)

The Setup: The robot is asked a tricky question with three options (A, B, or C) and has to pick one.
The Result: Again, the robots struggled. They picked answers almost randomly, getting about 33% right (which is what you get by guessing).
The Analogy: The robot didn't know how to play this specific game either. Because it was guessing blindly, it happened to pick the "stereotypical" answer about 50% of the time and the "anti-stereotypical" answer 50% of the time. This made it look like the robot was perfectly fair, but it was actually just clueless. The test couldn't tell if the robot was biased or not because it wasn't trying hard enough.

The Big Takeaway

The paper concludes with a very important lesson: You cannot judge a robot's fairness with just one test.

The "Logit" vs. The "Output": The researchers found that the privacy fog successfully "smoothed out" the robot's internal math (the probabilities it calculates in its brain). This made it less likely to think in stereotypes. However, this didn't always translate to the robot saying or doing fair things in the real world.
The Disconnect: Just because the robot's internal math is less biased doesn't mean its final output will be fair. It depends entirely on how you ask it to show its work.

In simple terms: Adding privacy protection (the fog) made the robot less likely to memorize and repeat stereotypes when taking a strict quiz. But when you asked it to write a story or solve a puzzle, the fog didn't automatically make it fair. To really know if an AI is fair, you have to test it in many different ways, not just one.

Technical Summary: How Does Differential Privacy Affect Social Bias in LLMs? A Systematic Evaluation

Problem Statement
Large Language Models (LLMs) trained on web-scale corpora risk memorizing sensitive training data, posing privacy threats. Differential Privacy (DP), specifically via DP-SGD, has emerged as a principled framework to limit the influence of individual data points during training, thereby reducing memorization. However, the relationship between limiting memorization through DP and the mitigation of social bias in LLMs remains poorly understood. While DP might weaken reliance on stereotypical associations by dampening the influence of specific training examples, the noise introduced during training could also distort learned representations unpredictably. Existing research has primarily explored this interaction in fine-tuning settings, leaving a gap in understanding how bias is shaped when DP is enforced during the pretraining phase itself.

Methodology
The authors present a systematic evaluation of social bias in a pretrained LLM trained with DP-SGD, comparing it against non-DP baselines. The study utilizes VaultGemma-1B, the only publicly available LLM pretrained entirely with DP-SGD (privacy budget $\epsilon = 2$ ), as the DP model. It is compared against two non-DP baselines: Gemma-3-1B-PT (matching the DP model in size) and Gemma-2-2B (matching the training family but larger).

To ensure a comprehensive assessment, the evaluation spans four complementary paradigms:

Sentence Scoring: Measures bias through controlled likelihood comparisons using datasets like StereoSet, WinoBias, and CrowS-Pairs. Metrics include Language Model Score (LMS), Stereotype Score (SS), and Idealized Context Association Test (ICAT).
Text Completion: Captures bias in open-ended generation using BOLD and HolisticBias. Evaluation focuses on regard, sentiment, and toxicity, calculating bias gaps between demographic groups.
Tabular Classification: Evaluates group fairness on text-serialized tabular data (Adult Census Income, COMPAS, German Credit) using In-Context Learning (ICL). The study tests four ICL strategies (Random, Balanced, Instruction, Removal) to isolate model-intrinsic bias from prompt-induced bias.
Question Answering: Assesses bias via multiple-choice questions with ambiguous context using the BBQ dataset, measuring accuracy on disambiguated contexts and Stereotype Scores on ambiguous ones.

Key Results
The study reveals a significant discrepancy between logit-level bias and output-level bias, with results varying substantially across evaluation paradigms:

Sentence Scoring: DP consistently reduces stereotypical bias. The VaultGemma-1B model achieved lower Stereotype Scores (SS) and higher ICAT scores compared to non-DP baselines, particularly in tasks requiring reliance on world knowledge (e.g., WinoBias Type 1). This suggests DP weakens the model's confidence in assigning higher likelihoods to stereotypical completions.
Text Completion: The benefits of DP observed in sentence scoring did not generalize reliably. While the DP model showed smaller bias gaps in regard and toxicity for certain demographics, results were mixed across metrics and datasets. For instance, non-DP models sometimes outperformed the DP model in sentiment bias gaps.
Tabular Classification: No consistent advantage for the DP model was observed. Performance across all models remained close to the random baseline (chance level), and fairness metrics were heavily influenced by the choice of in-context learning strategy rather than the model's pretraining privacy status. This indicates that pretrained base models struggle to extract predictive signals from structured tabular demonstrations via ICL.
Question Answering: On the BBQ benchmark, all models exhibited accuracy near the random baseline and SS values close to 50. The authors attribute this to the inability of base models (lacking instruction tuning) to reliably follow the multiple-choice task format, rendering the evaluation of intrinsic bias via this generation-based formulation ineffective.

Key Contributions

Unified Evaluation: This is the first work to systematically evaluate the effect of differentially private pretraining on social bias in generative LLMs across four distinct paradigms.
Paradigm Discrepancy: The authors demonstrate that while DP reduces stereotypical bias in sentence scoring (logit-level), this improvement does not consistently transfer to text completion, tabular classification, or question answering (output-level).
ICL Sensitivity: The study highlights that tabular classification results are highly sensitive to in-context learning strategies, limiting their utility for isolating model-intrinsic bias in pretrained base models.
Evaluation Limitations: The work underscores that conclusions about social bias vary substantially depending on the evaluation paradigm, revealing limitations in relying on single-benchmark assessments.

Significance and Claims
The paper claims that reducing memorization via DP does not necessarily reduce unfairness in all contexts. The findings highlight a critical disconnect: weakening stereotypical associations at the logit level does not guarantee equitable outputs in open-ended generation or decision-making tasks. Consequently, the authors argue that privacy-preserving training alone is insufficient to ensure equitable model outputs. They emphasize the necessity of multi-paradigm evaluation when assessing fairness in LLMs, as a single evaluation method may provide an incomplete or misleading picture of a model's behavior. The study concludes that privacy and fairness may require fundamentally different interventions, as the mechanisms that reduce memorization do not uniformly mitigate social bias across all task types.

1. The "Fill-in-the-Blank" Game (Sentence Scoring)

2. The "Story Writing" Game (Text Completion)

3. The "Data Table" Game (Tabular Classification)

4. The "Multiple Choice" Game (Question Answering)

The Big Takeaway

More like this