Shiny Stories, Hidden Struggles: Investigating the Representation of Disability Through the Lens of LLMs

Technical Summary: Shiny Stories, Hidden Struggles: Investigating the Representation of Disability Through the Lens of LLMs

Problem Statement
While Large Language Models (LLMs) have demonstrated the ability to simulate human personas and generate text reflecting diverse demographic groups, there is a critical concern regarding how these models represent historically marginalized communities. Prior research has extensively documented negative biases and stereotypes in LLMs. However, recent safety mechanisms, such as embedded AI guards and debiasing techniques, may lead to a different form of harm: "overcompensation." This phenomenon occurs when models, in an effort to avoid negative bias, generate overly positive, idealized, or sanitized portrayals of marginalized groups. This "toxic positivity" can erase the complexities, structural challenges, and negative emotional realities faced by these communities, effectively silencing their lived experiences under a veneer of inspiration. The paper specifically targets the representation of disability, a group that has received comparatively less attention in bias literature despite facing significant discrimination and socio-economic disadvantages.

Methodology
The authors employ a comparative analysis approach, contrasting self-descriptions from real individuals with disabilities against AI-generated portrayals. The study involves three primary datasets:

1. REDD (Real-World Dataset): A collection of 1,250 social media posts (Reddit) written by individuals self-identifying with disabilities. The data was sourced from six subreddits ($r/disability$, $r/blind$, $r/autism$, $r/depression$, $r/deaf$, $r/cerebralpalsy$). Posts were filtered to ensure first-person perspective and explicit mention of disability, then manually annotated by three human annotators (Fleiss' Kappa = 0.875).
2. LLMD (LLM-Disability Dataset): Artificial portrayals generated by three LLMs (GEMINI-1.5F, GPT-4O-MINI, and MIXTRAL-8B). Using open-ended prompts inspired by prior persona studies, the models were instructed to imagine they were individuals with specific disabilities (e.g., autism, blindness, depression) and write a social media post. This resulted in 1,080 posts (360 per model).
3. LLMND (LLM-Non-Disability Dataset): A control set of 1,080 posts generated by the same models simulating "generic" individuals without explicit mention of disability, used to isolate language shifts specific to the disability persona.

Analysis Metrics
The study utilizes a multi-dimensional quantitative and qualitative framework:

• Sentiment Analysis: Using VADER to classify posts as positive, negative, or neutral.
• Emotion Distribution: Employing the NRC Word-Emotion Association Lexicon (EmoLex) to quantify the prevalence of eight emotions (anger, fear, anticipation, trust, surprise, sadness, joy, disgust).
• Depression Detection: Applying a state-of-the-art model from the LT-EDI-ACL2022 Shared Task to predict depression levels (none, moderate, severe).
• Thematic and Lexical Analysis: Using "Fightin' Words" methodology (z-scores) to identify statistically significant distinguishing words between datasets. These words are then clustered into thematic groups using GPT-4O-MINI to interpret semantic differences.

Key Findings
The analysis yields two primary findings regarding the representation of disability:

1. Idealization and Toxic Positivity (RQ1): LLMs significantly diverge from the self-descriptions of real people with disabilities.

   • Sentiment: While 53.06% of real-world posts exhibited negative sentiment, LLM-generated posts were overwhelmingly positive (96.39% to 99.72%).
   • Depression: Real-world posts showed high rates of moderate (26.26%) and severe (20.42%) depression indicators. In contrast, LLMs almost entirely eliminated these signals; GPT-4O-MINI produced zero posts with depression indicators, and MIXTRAL-8B produced only one.
   • Lexical Differences: Real-world posts were distinguished by words related to medical struggles (pain, hospital, surgery), economic hardship (job, money, homeless), and negative emotional states (hate, stress, suicidal). LLM posts were dominated by words of inspiration and community (love, grateful, victory, strength, inclusive), reflecting a narrative of "overcoming" rather than living with disability.

2. Negative Bias and Oversimplification (RQ2): When comparing LLMs simulating disabled individuals versus generic individuals, the models exhibit a distinct shift in tone.

   • Emotional Tone: Despite the overall positivity, posts simulating disability contained significantly higher levels of negative emotions (anger, sadness, fear) and lower levels of positive anticipation and joy compared to posts simulating generic individuals.
   • Thematic Disparity: LLMs associate disability disproportionately with themes of struggle, resilience, and advocacy, while generic personas are associated with work, career, and leisure (entertainment, adventure). This suggests that LLMs frame disability primarily through a lens of "struggle and exceptionalism," neglecting other facets of human life such as professional achievement or recreational interests.

Significance and Contributions
The paper contributes a publicly available dataset (REDD and LLMD) and offers a critical examination of the "positive bias" in AI. The authors argue that the current trend of over-idealization in LLMs, likely driven by reinforcement learning from human feedback (RLHF) and safety guardrails, creates a "shiny" narrative that masks the "hidden struggles" of disability.

The significance of this work lies in its demonstration that bias mitigation in LLMs is not merely about removing negative stereotypes but also about avoiding "toxic positivity" that erases the reality of suffering and structural inequality. The authors posit that for AI to foster genuine inclusion and empathy, it must be capable of representing the full spectrum of human emotion, including pain, frustration, and depression, rather than forcing a sanitized, inspirational arc. The study underscores the need for AI policies and design practices that prioritize authentic representation over superficially benevolent portrayals, ensuring that technological tools align with the nuanced and often difficult lived realities of marginalized communities.