🤖 AI

Position: Fairness Failure in Generative Models is an Evaluation Problem

This position paper argues that fairness failures in generative models are primarily an evaluation problem caused by non-comparable and non-actionable findings, and proposes "Fairness Cards" as a standardized reporting artifact to enable reproducibility, comparability, and accountability in bias assessment.

Original authors: Mariia Vladimirova, Jean-Yves Franceschi, Thibaut Issenhuth

Published 2026-08-19

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mariia Vladimirova, Jean-Yves Franceschi, Thibaut Issenhuth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the last decade, computers have learned to create art, write stories, and hold conversations with a fluency that once seemed impossible. These generative systems, which can produce images, text, and video from simple instructions, are now woven into the fabric of daily life. Yet, as these tools become more powerful, a persistent shadow follows them: the tendency to reproduce and amplify societal inequalities. When a computer generates an image of a doctor, it often defaults to a man; when it writes a story about a leader, it frequently chooses a specific gender or race. This is not merely a glitch in the code but a reflection of the data the machines learned from, a mirror held up to human history's biases. For years, researchers and regulators have tried to fix this, developing methods to detect and reduce these unfair patterns. The goal has always been to ensure that these powerful tools treat all people with equal dignity and accuracy.

However, a new perspective suggests that the problem might not be the lack of better fixes, but rather the way we measure whether the fixes work. A team of researchers argues that the field is stuck in a cycle of confusion because the tests used to judge fairness are too flexible. Just as a ruler made of rubber would give different measurements depending on how you stretch it, the current methods for testing AI fairness change their results based on tiny, often unreported choices made by the researchers. One study might use a specific way of asking a question and conclude the AI is fair, while another study using a slightly different phrasing might conclude the same AI is deeply biased. Both studies could be technically correct within their own narrow rules, yet they tell opposite stories to the public and to policymakers. This inconsistency makes it impossible to know if the technology is actually improving or if we are just seeing different angles of the same problem.

The researchers behind this new work, Mariia Vladimirova and her colleagues, set out to diagnose why fairness in generative models remains such a stubborn issue. They did not simply propose a new algorithm to make AI more polite; instead, they examined the very process of evaluation itself. Through a series of controlled experiments, they demonstrated that the verdict on whether an AI is fair or unfair often depends entirely on the "protocol" used to test it. In one striking demonstration, they took a single, unchanging AI model and tested it against a grid of different scenarios. They varied the way questions were asked, the style of the answers expected, and the settings used to generate the text. They found that the same model could be judged as having a severe fairness problem under one set of conditions and as being perfectly balanced under another.

For instance, when the researchers asked the model to write a story about a person in a specific job, the results were heavily skewed toward stereotypes. But when they asked the same model to write a professional memo about that same person, the bias vanished. In another test, they changed the way the computer selected its words, a setting known as the decoding regime. A slight shift in this setting, which is often left to chance or default settings in other studies, was enough to flip the conclusion from "biased" to "unbiased." The researchers found that these variations were not random noise but systematic shifts that could hide or reveal harm depending on the choices made by the evaluator. This means that a company could release a model with a report claiming it is fair, simply because they chose a specific testing method that happened to show good results, while a different method would have revealed significant problems.

The paper argues that this lack of standardization is the true bottleneck. It is not that we do not know how to build fairer models, but that we cannot reliably tell if we have succeeded. The current landscape is filled with "ad-hoc" checks, where researchers pick a few examples to test and report the results. This approach allows for what the authors call "cherry-picking," where a system looks good because the test was designed to make it look good, or where a genuine improvement is missed because the test was too narrow. Furthermore, the researchers highlighted that modern AI systems often include safety layers that refuse to answer certain questions. If a model refuses to answer a question for one group of people but answers it for another, that is a form of unfairness called an access disparity. Yet, many current tests simply ignore these refusals or treat them as errors to be discarded, effectively hiding the fact that some people are being silenced while others are heard.

To solve this, the team proposes a new standard called a "Fairness Card." Imagine a nutrition label on a food package, but for AI systems. Just as a nutrition label tells you exactly what is in the food, how much sugar it contains, and how it was processed, a Fairness Card would list every single detail about how the AI was tested. It would specify exactly what questions were asked, how many times the test was run, what settings were used to generate the answers, and how the researchers handled refusals or safety blocks. This document would not dictate what "fair" means, but it would make the process of measuring fairness transparent and reproducible. If two different researchers want to compare their results, they would know exactly how to replicate the test, ensuring that they are comparing apples to apples rather than apples to oranges.

The researchers tested this idea by creating a detailed Fairness Card for their own experiment. They documented every variable, from the specific words used in the prompts to the random seeds that influenced the computer's choices. When they published these details, they showed that the same model could produce a wide range of fairness scores depending on which part of the card you looked at. Under one prompt style, the model showed a high rate of stereotypical language; under another, it showed almost none. By laying all these numbers out in a single, standardized format, they proved that the "truth" about the model's fairness is not a single number, but a complex picture that only becomes clear when all the variables are visible.

This approach shifts the focus from trying to find a perfect, one-size-fits-all definition of fairness to building a system where the evaluation itself is honest and complete. The authors acknowledge that this will not solve every problem. They note that some companies may still hide their data, and that different cultures may have different ideas about what is fair. However, they argue that without a shared, detailed record of how a system was tested, no progress can be made. It is impossible to improve what you cannot measure consistently, and it is impossible to compare solutions if the rules of the game keep changing.

The paper concludes with a call to action for researchers, developers, and regulators. They urge that any time a new AI system is released or a claim is made about its fairness, a Fairness Card must be provided. This document should be treated as a mandatory part of the release, just like a safety manual or a user guide. It should detail the specific groups of people tested, the exact methods used to ask questions, and the full range of results, including the worst-case scenarios. By making these choices explicit, the field can move away from vague promises and toward a future where fairness is a measurable, trackable, and accountable part of how these powerful tools are built and used. The goal is not to eliminate all bias, which may be impossible, but to ensure that we are all looking at the same evidence when we decide whether a system is safe to use.

Technical Summary: Fairness Failure in Generative Models is an Evaluation Problem

1. Problem Statement

Despite significant advancements in generative AI, concerns regarding fairness—specifically the reinforcement of societal inequalities and harm to marginalized groups—remain under-addressed. The authors argue that the persistence of fairness failures is not primarily due to a lack of mitigation techniques, but rather stems from a fundamental evaluation problem.

Current fairness assessments in generative models suffer from a lack of comparability and actionability. Empirical findings are rarely comparable across different papers or actionable for deployment decisions because they are highly sensitive to evaluation protocols. Small variations in prompt templates, paraphrasing, decoding/sampling settings, random seeds, and post-processing can materially alter measured demographic skews and stereotype scores. Consequently, two studies evaluating the same model can reach scientifically incompatible conclusions, both technically correct under their specific protocols. Furthermore, modern deployed systems introduce additional instability through safety layers (refusals and deflections) that shift access disparities, often rendering fairness a property of the served system rather than just the base model.

2. Methodology and Diagnosis

The paper diagnoses recurring empirical and conceptual failure modes in current generative fairness evaluation practices through a combination of literature review and controlled empirical audits.

2.1. Diagnostic Framework

The authors identify three primary drivers of evaluation instability:

Prompt Families and Generation Settings: Fairness outcomes are protocol-dependent. Variations in prompt wording, style, or context can induce divergent demographic distributions and sentiment.
Safety-Layer Behaviors: Refusals and deflections are often treated as missing data or noise, yet they constitute "access fairness" issues. Safety filters can introduce systematic disparities in who can access information or creative content based on protected attributes.
Scoring and Labeling Pipelines: Metrics are unstable and dependent on the choice of scoring models (automatic or human), rubrics, and annotator pools. Subjective labeling and imperfect automatic scorers further undermine reliability.

2.2. Empirical Audit (Qwen2.5-7B-Instruct)

To demonstrate these failure modes, the authors conducted a controlled audit on the Qwen2.5-7B-Instruct model.

Setup: The model was held fixed while the evaluation protocol varied across a full factorial grid:
- Demographic Slices: Four intersectional cells: {Male, Female} × {Christian, Muslim}.
- Prompt Families: Four distinct framing styles (F1: Job applicant description; F2: Story continuation; F3: Workplace incident bullets; F4: HR memo).
- Paraphrases: Five variations per family.
- Occupations: CEO, nurse, engineer, teacher.
- Decoding Regimes: Low entropy ( $t=0.2$ ) and high entropy ( $t=0.7$ ).
- Seeds: Five random seeds per configuration.
- Total: 3,200 generations.
Scoring: A deterministic lexical rubric measured refusal, deflection, stereotype keywords, demeaning language, identity salience, and title mentions.

2.3. Additional Probes

The authors also included qualitative probes on Stable Diffusion (image generation) and Mistral (conversational agents) to illustrate that bias persists despite mitigation techniques and that model behavior can remain stereotype-consistent even with disclaimers.

3. Key Contributions

3.1. The "Fairness Card"

The primary contribution is the proposal of Fairness Cards, a minimal, standardized reporting artifact designed to make fairness evaluation auditable, explicit, and reproducible. Unlike existing documentation (e.g., Model Cards) which often underspecify the evaluation protocol, Fairness Cards treat the evaluation procedure itself as a first-class disclosure object.

Mandatory Fields for a Fairness Card:

Model & System Identification: Version, modality, deployment surface, and differences between base and served systems.
Intended Use & Context: Target users and high-risk contexts.
Fairness Scope & Harm Model: Specific harms in scope (representational, allocative, access) and perspective.
Protected Attributes & Slices: Operationalization of attributes and required intersections.
Evaluation Protocol: Explicit details on prompt families, paraphrase strategies, counterfactual swaps, seed policies, decoding settings, and refusal handling rules.
Metrics & Decision Rules: Scoring models, annotator rubrics, and thresholds for pass/fail decisions.
Mitigations & Trade-offs: Interventions applied and measured trade-offs between fairness and utility.
Governance & Monitoring: Plans for post-release monitoring and versioning.

3.2. Modality-Specific Extensions

The paper outlines how Fairness Cards should be adapted for different modalities:

Image: Focus on visual stereotyping, proxy attributes, and sexualization.
Text (LLM): Focus on refusal/deflection disparities, tone, and discursive framing.
Audio/Speech: Focus on accent stereotyping and intelligibility.
Video: Focus on agency, role persistence, and narrative stereotypes.
Multimodal: Focus on cross-modal consistency and compounded biases.

4. Key Results

4.1. Protocol-Dependent Verdicts

The Qwen2.5 audit demonstrated that the same model can receive opposite fairness verdicts depending solely on the evaluation protocol:

Prompt Family Sensitivity: Under prompt family F2 (story continuation), the worst-slice stereotype-keyword rate exceeded 0.23 (23%), triggering a "flagged" status. Under F4 (HR memo), the same slice's rate dropped to 0.04 (4%), resulting in a "not flagged" status.
Seed and Decoding Instability: Even with fixed prompts, varying random seeds shifted the worst-slice stereotype rate from 0.094 to 0.125, enough to flip a near-threshold judgment.
Refusal Handling: In this specific audit, refusal rates were negligible, but the authors note that in other contexts, excluding refusals from evaluation hides significant access disparities.

4.2. Failure of Current Documentation

Analysis of recent technical reports (e.g., Mixtral, GPT-5, Llama 3) revealed that while bias is often acknowledged, the evidence provided is rarely comparable due to underspecified protocols. Many reports lack counterfactual, intersectional, or refusal reporting, making it impossible to determine if improvements are robust or merely artifacts of specific prompt choices.

5. Significance and Claims

The paper argues that the field of generative AI fairness cannot progress reliably until evaluation practices shift from ad-hoc bias checks to standardized, generative-specific evaluation.

Cumulative Evidence: Without standardized protocols, fairness evidence is non-cumulative. Researchers cannot reliably tell if a claimed mitigation improves fairness, merely changes the prompts/metrics, or shifts harms elsewhere (e.g., from biased content to refusals).
Decision Relevance: Fairness Cards enable stakeholders (developers, auditors, regulators) to make deployment decisions based on comparable, reproducible evidence rather than isolated, potentially cherry-picked results.
Not a Normative Solution: The authors explicitly state that Fairness Cards do not define fairness itself or resolve normative disagreements about which slices or harms matter. Instead, they standardize the disclosure of evaluation choices, making assumptions and degrees of freedom visible. This allows for "compliance theater" to be replaced by transparent, auditable claims.

The paper concludes that while improved data, training, and governance are necessary, they are insufficient without standardized evaluation to measure their impact. The proposed paradigm shift aims to make fairness a performance-critical dimension of generative systems, integrated throughout the model lifecycle.