💻 computer science

Bias at the End of the Score

This paper presents a large-scale audit revealing that reward models used in text-to-image generation inherently encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female subjects, reinforce stereotypes, and reduce demographic diversity.

Original authors: Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very strict, very fast art critic to judge thousands of paintings. You tell this critic, "I want the best possible art." The critic looks at the paintings and gives them a score from 1 to 10.

Now, imagine you use this critic's scores to teach an AI artist how to paint. Every time the AI paints something the critic likes, it gets a gold star. Every time it paints something the critic dislikes, it gets a red X. Eventually, the AI learns to paint only what the critic thinks is "good."

This paper is a big investigation into what happens when we use these "AI Critics" (called Reward Models) to train and judge AI art generators. The researchers found that these critics aren't actually neutral judges of quality. Instead, they have hidden, biased preferences that change the art in dangerous and unfair ways.

Here is the breakdown of their findings using simple analogies:

1. The "Gold Star" Problem: How the AI Gets Twisted

The researchers tested what happens when they let the AI optimize its paintings to get the highest possible score from these critics.

The "Strip-Tease" Effect: When the AI tried to get a higher score, it started making women in the images look increasingly sexualized. It was like the critic was secretly whispering, "Show more skin, and I'll give you a better grade."
- The Result: The AI started "undressing" female characters in the images, turning them into hyper-sexualized versions, even when the user just asked for a normal photo. Men in the images didn't get this treatment nearly as much.
The "White-Washing" Effect: When the AI tried to get a better score, it started changing the race of the people in the pictures.
- The Result: If you asked for a photo of a "doctor" without specifying a race, the AI would often start with a Black or Asian person and then, while trying to "improve" the image, slowly morph them into a White person. It's as if the critic believes a "good" doctor must look White.

2. Why is this happening? (The "Echo Chamber" Analogy)

You might think, "But the critic is just judging the quality of the picture, right? Like how sharp the focus is or how nice the colors are."

The paper says no. The critics are judging based on stereotypes they learned from their training data.

The "Popularity Contest" Analogy: Imagine the critic was trained by asking 1,000 people, "Which of these two photos looks better?"
- If the people answering the question subconsciously prefer White faces or sexualized women, the critic learns that "White" and "Sexy" = "High Score."
- The critic isn't measuring "artistic quality"; it's measuring "how well this image fits what society usually expects."
The "Occupation" Trap: The researchers found that when they asked for a "nurse," the critic gave higher scores to female faces. When they asked for a "CEO," it gave higher scores to male faces. The critic wasn't judging the photo's quality; it was judging whether the photo matched the real-world stereotype of that job.

3. The "Filter" That Skews Reality

The paper explains that these Reward Models are used everywhere in AI:

To filter out "bad" images before showing them to you.
To teach the AI how to draw better.
To decide which images are "safe."

Because these models are biased, they act like a funhouse mirror that only reflects a specific, narrow version of reality.

If you use them to filter images, you will never see a Black CEO or a sexualized man, because the "filter" deletes them before you see them.
If you use them to train the AI, the AI will forget how to draw diverse people and will only know how to draw the "stereotypical" version.

The Big Takeaway

The authors are saying: "We thought these AI critics were neutral judges of quality, but they are actually biased gatekeepers."

They are reinforcing harmful stereotypes (like women being sexual objects or certain races being less competent) and making the AI world less diverse. If we keep using these biased critics to train our AI, we aren't building a smarter future; we are just building a future that looks exactly like our worst stereotypes.

The Solution? We need to build new critics that are trained to ignore these stereotypes and judge images based on actual quality, not on who the person in the picture looks like.

1. Problem Statement

Reward Models (RMs) have become integral to the Text-to-Image (T2I) pipeline, serving as scoring functions for dataset filtering, evaluation metrics, and supervisory signals for fine-tuning (e.g., via Reinforcement Learning or test-time optimization). While RMs are designed to measure image quality, alignment, and human preference, their robustness and fairness regarding demographic biases remain largely unexamined.

The authors hypothesize that RMs are not neutral quality metrics but rather encode demographic priors learned from their training data (human preferences, synthetic datasets). When used to optimize T2I outputs, these biases may lead to:

Hypersexualization: Disproportionate sexualization of specific demographic groups.
Demographic Convergence: The erosion of diversity, where optimized images converge toward a dominant demographic (e.g., White, male).
Reinforcement of Stereotypes: Scoring systems that align with real-world occupational and social biases rather than objective image quality.

2. Methodology

The study employs a two-part audit framework to investigate RM biases during both optimization (changing the image) and evaluation (scoring the image).

Part I: Reward-Guided Optimization (The "Action" Phase)

Framework: The authors use the ReNO (Reward-based Noise Optimization) framework. They optimize the initial noise vector ( $\epsilon_0$ ) of a T2I model (e.g., SDXL-Turbo, PixArt- $\alpha$ ) using gradient ascent to maximize the RM score, while keeping the model parameters frozen.
Objective: To observe how optimizing for "quality" alters the generated image's demographic attributes when the prompt is underspecified (e.g., "a photo of a doctor").
Metrics:
- Hypersexualization: Measured via changes in NSFW classification rates and Skin Exposure (SE) ratios (proportion of visible body skin).
- Demographic Drift: Using a Chicago Face Database (CFD) anchor-based approach, they classify the perceived race and gender of generated images before and after optimization to track shifts (e.g., Black $\to$ White, Female $\to$ Male).
Baselines: Includes an "Incompression" baseline (maximizing high-frequency DCT coefficients) to distinguish between data-driven bias and inherent model optimization artifacts.

Part II: Counterfactual Evaluation (The "Scoring" Phase)

Datasets: Three counterfactual datasets (CausalFace, SocialCounterfactuals, PAIRS) containing image pairs that differ only in demographic attributes (race/gender) while holding semantic context constant.
Prompts: Four categories of prompts: Stereotype Content Model (SCM), ABC model (Agency/Belief), DALL-Eval descriptors, and Occupation sets.
Analysis:
- Linear Regression: Fits an OLS model to predict RM scores based on race ( $\rho$ ) and gender ( $\gamma$ ) to quantify systematic score disparities ( $\beta$ coefficients).
- Ranking Analysis: Computes the average rank of demographic groups to identify consistent preference ordering.
- Correlation: Correlates RM score differences with real-world labor statistics (U.S. Bureau of Labor Statistics) to test if RMs encode occupational gender distributions.

3. Key Contributions

Discovery of Optimization-Induced Bias: Demonstrated that using RMs to optimize T2I outputs causes systematic, non-uniform demographic shifts, specifically hypersexualizing female subjects and converting non-White subjects to White.
Scoring-Level Mechanism Analysis: Proved that RM scores are significantly predicted by demographic attributes, mirroring real-world societal biases (e.g., occupational gender distributions) rather than objective image quality.
Comprehensive Audit: Evaluated five widely used RMs (PickScore, ImageReward, HPS, VQAScore, CLIP) across multiple base models and datasets, providing a large-scale quantitative audit of the T2I reward ecosystem.
Methodological Framework: Established a rigorous protocol for auditing RMs using counterfactual datasets and noise optimization, separating the effects of the generative model from the reward signal.

4. Key Results

A. Disparate Hypersexualization

PickScore exhibited the strongest bias, increasing the NSFW classification rate by 19% for female subjects versus 7% for male subjects (a 2.7 $\times$ disparity).
Optimization led to increased skin exposure and undressing, disproportionately affecting female subjects across all tested base models.

B. Demographic Convergence (Race and Gender)

Race Convergence: When optimizing with ImageReward and HPS, >80% of images initially classified as Black subjects were reclassified as White after optimization. Non-White subjects rarely remained non-White, whereas White subjects remained White.
Gender Convergence: Optimization with ImageReward and CLIP caused female subjects to reclassify as male in 39% and 26% of cases, respectively.
Directionality: These shifts occurred even when the prompt contained no demographic information (e.g., "a photo of a doctor"), indicating the RM forces the latent space toward a "default" (White, male) representation.

C. Scoring Disparities and Real-World Correlation

Ranking Bias: HPS and ImageReward consistently ranked White subjects highest and Asian subjects lowest, regardless of whether the prompt was positive or negative.
Occupational Bias: RM scores correlated strongly with real-world gender employment statistics. RMs scored male faces higher for male-dominated occupations and female faces higher for female-dominated ones.
Interaction Effects: In negative prompts (e.g., "dishonest," "emotional"), the preference for White subjects persisted, suggesting a structural prior over demographic identity rather than a reaction to semantic content.

5. Significance and Implications

Reliability of RMs: The findings challenge the reliability of current RMs as neutral quality metrics. They act as demographic filters that implicitly enforce societal biases.
Pipeline Risks: Any T2I pipeline using these RMs for Best-of-N selection, fine-tuning, or dataset curation will inadvertently amplify demographic homogeneity and harmful stereotypes (e.g., sexualizing women, erasing racial diversity).
Future Directions: The paper calls for improved data collection (diverse human preferences), training procedures that decouple quality from demographic priors, and the development of robust, bias-aware scoring mechanisms.

In conclusion, the paper argues that "bias at the end of the score" is a critical failure mode in generative AI, where the very tools designed to improve image quality are systematically degrading demographic diversity and reinforcing harmful societal stereotypes.