Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed

Technical Summary: Harm is Not Universal: Community-Specific Toxicity Detection is Urgently Needed

Problem Statement
Current state-of-the-art (SoTA) toxicity detectors for text-to-image (T2I) generation employ a "one-size-fits-all" approach, utilizing a single universal model trained on fixed safety guidelines (e.g., hate speech, violence, sexual content) to protect all users. The authors argue that this paradigm fails to shield marginalized communities from representational harms—subtle, culturally specific misrepresentations that arise from limited or biased training data rather than overt toxicity. Empirical evidence presented in the paper indicates that approximately 35% of images labeled "safe" by SoTA detectors are considered harmful by disability communities. These harms, such as depicting wheelchair users with wheels instead of legs or portraying blind individuals with blindfolds during daily activities, are often hallucinated by models and lack real-world counterparts, rendering standard detectors ineffective.

Methodology
To address this gap, the authors propose Community-Specific Toxicity Detection (CTD), a framework that extends traditional toxicity detection to account for localized representational harms. The methodology involves:

1. Guideline Development: Collaborating with disability experts and advocacy organizations to define safety guidelines for two specific communities: people who are blind or have low vision (BLV) and people with dwarfism (DWF). These guidelines identify six specific representational harms per community (e.g., non-anatomical distortions, infantilization, inappropriate assistive technology usage).
2. Dataset Curation: Generating 2,400 T2I images (1,200 per community) using community-authored prompts and three T2I models (GPT-Image-1, Imagen 4-Ultra, Stable Diffusion 3.5). Five disability experts annotated these images for safety, achieving high inter-rater reliability (Cohen's Kappa of 0.65 for BLV and 0.97 for DWF).
3. Experimental Evaluation: The authors evaluated three categories of models under zero-shot, inference-time adaptation, and fine-tuning settings:
   • General-purpose Vision-Language Models (VLMs): Including GPT-5, GPT-4o, LLaVA, Qwen, and LLaMA.
   • Safety-Specialized Detectors: Models like LlavaGuard, QwenGuard, and ShieldGemma2 trained on general public safety taxonomies.
   • Adaptation Strategies:
     • Zero-shot: Providing community guidelines at inference without training.
     • In-Context Learning (ICL): Providing 2–5 annotated examples of harmful content.
     • Visual Question Answering (VQA): Framing safety evaluation as a series of targeted yes/no questions derived from guidelines.
     • Parameter-Efficient Fine-Tuning (FT): Using LoRA to fine-tune smaller models (0.5B–7B parameters) on fewer than 100 annotated examples.

Key Results
The study yields several critical findings regarding the limitations of current systems and the potential of CTD:

• Universal Detectors Fail: In zero-shot settings, both general-purpose VLMs and safety-specialized detectors catastrophically fail to recognize community-specific harms. Most models achieved F1 scores near zero (e.g., 0.00 for many models), and even the best performers (GPT-4o) scored below 0.35 for BLV. This performance is often worse than random guessing baselines.
• Adaptation Improves Performance:
  • VQA: This approach yielded the most consistent gains, with GPT-4o achieving F1 scores of 0.78 for DWF and 0.49 for BLV. VQA proved effective even for smaller models.
  • ICL: Showed substantial but inconsistent improvements. While it improved grounding for some models, it failed to consistently improve performance across all communities (e.g., performance dropped for DWF with some models).
  • Fine-Tuning: Parameter-efficient fine-tuning (LoRA) on small datasets (<100 examples) significantly improved smaller VLMs, raising F1 scores to 0.59 for Qwen-7b on DWF and 0.48 for LLaVA-7b on BLV. However, these scores remain well below the F1 ≈0.9 typically achieved for general-purpose toxicity detection.
• Sensitivity to Evolving Guidelines: The authors conducted ablation studies simulating the addition or removal of harm definitions. Fine-tuned models showed limited flexibility; while they adapted partially, their performance on other harm categories shifted significantly (relative changes of ~20%), suggesting that static fine-tuning may require frequent retraining to align with evolving community norms.

Significance and Claims
The paper positions itself as a call to action for the AI safety research community, arguing that the "universal" safety paradigm is insufficient for protecting marginalized groups. The authors claim that:

1. CTD is Urgent: Current detectors leave marginalized communities vulnerable to material consequences, such as reduced access to education and employment, due to the reinforcement of stigma and distorted public perceptions.
2. CTD is Feasible but Challenging: While CTD is technically feasible using expert-guided guidelines and adaptation strategies, it presents distinct challenges compared to general toxicity detection, including the need for dynamic guidelines, the coexistence of multiple community definitions, and extreme data scarcity.
3. Research Directions: The paper identifies four critical research questions: how to learn from scarce data, how to adapt to dynamically evolving guidelines, how to architect systems for multiple non-aggregatable guidelines, and how to build infrastructure for distributed community-researcher partnerships.

The authors emphasize that achieving inclusive AI safety requires sustained engagement and trust-building with diverse advocacy organizations, a process that cannot be accelerated by engineering alone. They conclude that while their work demonstrates the promise of CTD, significant research effort is needed to bridge the performance gap between community-specific detection and general-purpose safety standards.