💻 computer science

Beyond wheelchairs and blindfolds: Investigating disability stereotypes in T2I models with INCLUDE-BENCH

This paper introduces INCLUDE-BENCH, the first large-scale benchmark for evaluating disability-related biases in text-to-image models, revealing that current systems systematically rely on narrow stereotypes like wheelchairs and exhibit stronger alignment with real-world prejudicial associations when depicting people with disabilities.

Original authors: Sophia Lichtenberg, Albert Gatt, Judith Masthoff

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sophia Lichtenberg, Albert Gatt, Judith Masthoff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical art machine that can draw whatever you describe. You say, "Draw a chef," and it spits out a picture of a person in a white hat cooking. You say, "Draw a person with a disability," and... well, that's where the magic gets a little glitchy.

A team of researchers at Utrecht University decided to test this glitch with a new, giant test called INCLUDE-BENCH. They didn't just want to see if the machines were "biased"; they wanted to see how they were biased, using a massive dataset of 119,680 generated images. They asked 17 different art engines (both open-source ones anyone can use and two closed, secret ones) to draw people with disabilities in all sorts of situations.

Here is what they found, served up with a side of reality checks.

The "Wheelchair" Shortcut

The biggest surprise? The machines are obsessed with wheelchairs. When the researchers asked for a "mobility-impaired" person or just a generic "disabled" person, almost every single model immediately reached for the wheelchair. It's like if you asked a chef to draw a "baker," and they only drew someone holding a rolling pin, ignoring the ovens, the flour, or the actual bread.

The paper shows that for these specific prompts, the machines are incredibly consistent at drawing the wheelchair, but they are terrible at drawing anything else. It's a visual shortcut. The machines think, "Oh, disability? That means wheelchair," and they stop there.

The "Old and Sad" Filter

The machines also seem to have a weird filter for age and mood. When they drew people with mobility issues, they were overwhelmingly older. If the person was a woman, she was often drawn in a home, cooking or cleaning. If the person was a man, he was often outside, in a park or a museum.

The researchers found that the machines are basically playing a game of "stereotype bingo." They are picking the most obvious, recognizable clues (like a wheelchair or a blindfold) to make sure you know who the person is, but in doing so, they erase all the other details. They aren't showing a diverse group of people living their lives; they are showing a very narrow, very old, and very specific set of characters.

The "Alignment vs. Diversity" Trade-off

Here is a fun way to think about the math behind the magic. The researchers measured two things:

Alignment: How well does the picture match the words you typed?
Diversity: How different are the pictures from each other?

They found a funny trade-off. The pictures that matched the words "mobility impaired" or "disabled" the best (highest alignment) were the least diverse. They all looked almost exactly the same: an old person in a wheelchair.

But when they asked for "blind" or "deaf" people, the pictures were a bit more diverse. You might see a person with a cane, or someone wearing sunglasses, or maybe just someone looking thoughtful. But even then, the machines kept falling back on old tricks, like drawing blindfolds or specific hand gestures.

The "Warmth and Competence" Score

The researchers even invented a new way to score the pictures, called the SCM Score. Imagine a graph where one side is "How nice does this person seem?" (Warmth) and the other is "How capable do they seem?" (Competence).

In the real world, people often think of disabled folks as "nice but not very capable." The paper found that the art machines are copying this exact vibe. The pictures of people in wheelchairs scored low on "competence." They looked less capable than the pictures of people without disabilities. Even when the researchers asked the machines to draw these people doing active things (like walking or chatting), the machines still struggled to make them look capable.

What the Machines Got Wrong (and What They Didn't Test)

It's important to know what this test didn't do. The researchers explicitly said they did not test for cognitive or intellectual disabilities. Why? Because those aren't always visible in a picture. You can't really "draw" a thinking process the same way you can draw a wheelchair. So, the results we have are strictly about things you can see.

Also, the paper doesn't claim to have "fixed" the problem. They didn't find a magic switch to turn off the bias. Instead, they built a giant measuring tape (INCLUDE-BENCH) to show us exactly how broken the tape measure is right now. They found that even when they tried to change the context (asking for a person in a cafe vs. a park), the machines still leaned heavily on the stereotypes.

The Bottom Line

The paper suggests that these art machines aren't just accidentally making mistakes; they are actively reproducing the same narrow, stereotypical stories we see in the real world. They are over-weighting the "diagnostic" clues (the wheelchair, the cane) and ignoring the rest of the human being.

The researchers are pretty sure about this because they generated nearly 120,000 images and checked them with math and AI tools. They aren't guessing; they measured it. And the measurement says: if you want to see a real, diverse, capable person with a disability, you can't just ask the machine yet. The machine is still stuck on the "wheelchair and old age" loop.

So, while the technology is amazing, it's still learning that a person is more than just their disability. Until then, the magic art machine is mostly just drawing the same old story, over and over again.

Technical Summary: INCLUDE-BENCH – Investigating Disability Stereotypes in T2I Models

Problem Statement
While Text-to-Image (T2I) models have demonstrated significant capability in content creation, they are known to reproduce and amplify societal biases present in their training data. Prior research has extensively evaluated biases related to gender, race, age, and cultural representation, often within occupational or scenario-based contexts. However, disability remains systematically underexplored in this domain. Existing evaluation practices frequently fail to align with sociologically grounded definitions of stereotyping, limiting principled assessments of representational harms toward People with Disabilities (PWD). Current literature suggests T2I models often produce narrow, stereotypical portrayals of PWD—frequently depicting them as older, sad, and reliant on outdated aids like manual wheelchairs—while failing to represent the diversity of the disability community, which comprises an estimated 16% of the global population.

Methodology: INCLUDE-BENCH
To address these gaps, the authors introduce INCLUDE-BENCH (INCLUsive Disability Evaluation), the first large-scale benchmark designed to evaluate disability-related bias in T2I models. The methodology involves a structured pipeline of dataset construction, image generation, and multimodal evaluation.

Dataset Construction: The benchmark consists of 352 prompts organized into four subsets to isolate the effects of impairment, intersectional identity, and context:
1. No Context: Baseline representations without environmental cues.
2. No Context + Bias: Introduces intersectional attributes (gender, race, age).
3. Only Context: Introduces static spatial settings (e.g., museum, home, park).
4. Action: Grounded in the WHO International Classification of Functioning, Disability, and Health (ICF), these prompts test agentic representations in domains like mobility, communication, and self-care.
  The prompts vary along three axes: impairment specificity (generic, mobility, sensory), intersectional attributes, and environmental context. The study focuses on impairments with observable visual characteristics (e.g., Blind, Deaf, Mobility Impaired, Deafblind, Mute), excluding cognitive disabilities that are less visually apparent.
Image Generation: The authors generated 119,680 images by running the 352 prompts across 17 state-of-the-art T2I models (including 15 open-source models from the Stable Diffusion, FLUX, and other families, and 2 closed models). For each prompt, 20 images were generated using random seeds.
Evaluation Metrics:
- Visual Representation Analysis: Generated images were processed using SAM (Segment Anything Model) to crop person-centric regions. These were clustered using MiniBatchKMeans to identify recurring visual patterns. Image captions were generated using Qwen3-VL to characterize clusters and identify shared visual cues via token frequency.
- Intersectional Analysis: Demographic attributes (race, age, gender) were extracted via Visual Question Answering (VQA). The authors computed relative appearance ratios ( $R_{i,g}$ ) comparing the frequency of specific identities in impairment groups against a non-impairment baseline.
- Disability Semantic Alignment: Measured using CLIPScore to determine the strength of semantic alignment between generated images and disability-conditioned prompts.
- Spectral Image Diversity: Quantified using the Vendi Score (based on CLIP embeddings) to measure the diversity of unique visual elements within a set of generated images for a given prompt.
- Stereotype Content Model (SCM) Score: A novel metric introduced to link visual representations to the sociological Stereotype Content Model. This model assesses perceptions along two dimensions: Warmth (sociability, morality) and Competence (ability, agency). The score calculates the alignment of image embeddings with positive and negative textual vectors for these dimensions.

Key Results
The evaluation of 15 open-source and 2 closed models yielded four primary findings:

Dominance of Wheelchair Depictions: Mobility-impaired and default disability prompts predominantly yield wheelchair depictions across all models. This visual shorthand is the most common diagnostic feature, leading to a significant reduction in representational diversity.
Alignment-Diversity Trade-off: There is a clear inverse relationship between semantic alignment and visual diversity. Physical and generic disability prompts (e.g., "Mobility Impaired," "Disabled") achieve the highest CLIPScores (strongest alignment with the text) but exhibit the lowest Vendi Scores (lowest diversity). Conversely, sensory disabilities show moderate alignment but higher diversity, particularly in communicative and cognitive tasks.
Demographic Skew and Intersectional Bias: The models systematically overrepresent older white individuals, particularly older white women in mobility and domestic contexts, and older white men in public or active contexts. Younger individuals and non-white demographics are significantly underrepresented in disability-conditioned generations compared to the baseline.
Reproduction of Stereotypes via SCM: The introduced SCM Score reveals that T2I models reflect real-world stereotypical associations. Images associated with mobility impairments and wheelchairs are perceived as lower in both warmth and competence compared to non-impaired baselines. Interestingly, sensory ("invisible") disabilities were depicted with higher competence and warmth than mobility impairments, though still lower than non-disabled depictions.

Significance and Contributions
The paper claims the following contributions:

INCLUDE-BENCH: The creation of a comprehensive, large-scale benchmark comprising 119K images that evaluates disability bias across diverse static and dynamic contexts, addressing the lack of contextual evaluation in prior work.
Empirical Evidence of Bias: The study provides robust evidence that T2I models rely on narrow, highly diagnostic visual features (like wheelchairs and blindfolds) to maximize label alignment, resulting in stereotypical and demographically homogeneous portrayals.
Methodological Innovation: The introduction of the SCM Score offers a principled, multimodal method for quantifying how T2I models encode implicit social judgments regarding warmth and competence, linking technical outputs to sociological frameworks.
Call for Structural Change: The authors conclude that these biases are structurally embedded across different model architectures. They argue that current T2I systems do not merely reflect disability but reproduce simplified, socially stratified narratives. The paper underscores the necessity for bias-aware design and evaluation to achieve fairer visual representations, noting that current mitigation strategies yield only limited improvements in diversity or stereotype reduction.

The study maintains a modest scope, acknowledging limitations such as the reliance on automated metrics (which may inherit their own biases), the exclusion of human annotation by PWD due to scale, and the focus on visually apparent disabilities. Future work is suggested to involve PWD-centered evaluation to ensure findings align with the community's perceptions of authentic representation.