Technical Summary: Going PLACES

Problem Statement

Despite the global deployment of Text-to-Image (T2I) models, their safety frameworks remain largely calibrated to a Western-centric default. This "single story" approach creates significant vulnerabilities for the Global South, failing to capture local context-specific harms unique to the regions' social, cultural, and linguistic diversity. Existing red-teaming methods, whether conducted by developers or via open crowdsourcing, often reflect a geographical bias or produce data tethered to popular Western perspectives on safety. Consequently, existing safety frameworks miss unique adversarial patterns enabled by local cultural and linguistic nuances, such as code-mixing, and fail to identify novel harms like normative dissonance (e.g., violating religious norms or ignoring local customs).

Methodology

The authors present PLACES (Participatory Localized Red Teaming for Text-to-Image Safety in the Global South), a dataset and methodology designed to bridge the gap between large-scale crowdsourcing and deep cultural contextualization. The approach is defined by two core pillars: Localization and Participation.

1. Participatory Community Engagement:

   • Partnerships: The study established contractual partnerships with four universities in secondary urban centers (avoiding major metropolitan hubs) across the Global South: St. Joseph Engineering College (Mangalore, Karnataka, India), Lovely Professional University (Phagwara, Punjab, India), Bayero University Kano (Kano, Nigeria), and the University of Cape Coast (Cape Coast, Ghana).
   • Protocol: The team recruited university faculty as "challenge supervisors" and students as red teaming participants. Unlike standard crowdsourcing, this involved educational workshops on AI safety, adversarial testing, and local norms. Supervisors acted as participatory mediators, guiding participants to create prompts based on local metaphors, cultural symbols, and lived experiences.
   • Scaffolded Monitoring: A system of daily guidance and weekly feedback loops ensured participants could tap into regional surroundings and preferred conversational languages (including code-mixing) to surface contextualized harms.

2. Data Collection and Analysis:

   • Dataset: Over 12 weeks, participants generated adversarial prompts against a mix of T2I models (including StableDiffusion, Imagen, and MUSE). The resulting PLACES dataset contains 26,139 prompt-image pairs flagged as harmful, accompanied by human annotations explaining the harm, attack mode, and targeted identity.
   • Comparative Analysis: The study compares PLACES against the geography-agnostic Adversarial Nibbler (AdvNib) dataset to highlight differences in coverage and diversity.
   • Analytical Techniques:
     • UMAP Visualization: Used to project prompt embeddings and identify clustering patterns, revealing distinct region-specific themes.
     • Cultural Artifact Taxonomy: Utilized the Ethnographic Thesaurus to categorize culturally specific references (e.g., foodways, ritual, material culture) in prompts, distinguishing them from generic terms.
     • Linguistic Analysis: Investigated code-mixing patterns and their impact on safety filter bypassing using models like ShieldGemma and Perspective API.
     • Qualitative Discovery: Employed an LLM-in-the-loop approach to cluster free-text feedback from participants, followed by human thematic coding to uncover novel harm categories not covered by standard taxonomies.

Key Contributions

1. The PLACES Dataset

A novel red-teaming dataset comprising ~26,000 adversarial prompt-image pairs with human annotations. The dataset is characterized by:

• High Diversity: It captures a wide range of socio-cultural and linguistic attributes, including significant code-mixing (blending local languages with English).
• Novelty: Only 83 prompts overlap with the original Adversarial Nibbler collection, demonstrating that the participatory approach yields largely unique failure modes.
• Granularity: Includes specific annotations for attack modes, harm types, and targeted identities, revealing regional variations (e.g., higher prevalence of "Hate & Harassment" in Nigeria vs. "Religion" in India).

2. Participatory and Localized Methodology

The paper establishes a successful protocol for community-centered red teaming that:

• Prioritizes secondary urban centers to capture under-represented user realities.
• Incorporates contextualization practices (workshops, supervisor mediation) to surface region-specific insights.
• Demonstrates that engaging community members from distinct cultural contexts increases diversity beyond simply collecting more prompts, yielding novel insights without compromising scale.

3. Novel Safety Insights

The analysis reveals three critical findings regarding T2I safety in the Global South:

• Unique Adversarial Patterns: Localized cultural and linguistic nuances (specifically code-mixing) create adversarial vectors that non-localized approaches miss. For instance, code-mixed prompts were found to bypass safety filters (ShieldGemma) that flagged the English-only translations as unsafe.
• Normative Dissonance: The study identifies "normative dissonance," where model outputs violate local moral boundaries that do not align with Western safety guidelines. Examples include:
  • Violating religious norms (e.g., depicting a Hindu man eating beef, Muslims gambling in Mecca).
  • Ignoring local laws and customs (e.g., depicting peacocks as pets in India, chewing gum in Singapore).
  • Rendering cultural taboos (e.g., black cats crossing roads, specific days for fishing).
• Representational Harms: The paper details specific failures in representation, including:
  • Ontological Flattening: Collapsing complex, region-specific concepts into Western equivalents (e.g., replacing regional dishes with burgers, depicting "Yakshagana" artists with generic classical dance costumes).
  • Cultural Erasure: Replacing distinct visual markers of identity with generic or incorrect tropes (e.g., generating a white man performing surgery for a prompt about an African surgeon).
  • Demeaning Depictions: Systematic association of local populations with poverty or violence, even when prompts imply high socioeconomic status.

Results and Significance

The paper argues that expanding T2I safety requires moving beyond mere scale to incorporate deeply localized, participatory methodologies. The results substantiate that:

• Safety is Context-Dependent: Model failures arise as a function of different categories of cultural and linguistic cues. What is considered "safe" or "harmful" is deeply rooted in local norms.
• Limitations of Current Frameworks: Existing safety frameworks and classification models (like ShieldGemma) show low agreement with human-annotated harms in the Global South, particularly regarding "Stereotypes & Bias" and "Other" harm categories.
• The Value of Participation: By empowering community members to define harms based on their lived experiences, the study uncovered structural gaps in safety frameworks that top-down or geography-agnostic approaches failed to detect.

The significance of this work lies in its demonstration that truly global AI systems must bridge the gap between large-scale crowdsourcing and localised community engagement. The PLACES dataset and methodology provide a rich basis for furthering AI safety evaluations, specifically focusing on the under-represented Global South, and highlight the necessity of addressing normative dissonance in future model alignment.