InsideOut: Measuring and Mitigating Insider-Outsider Bias in Interview Script Generation

1. Problem Definition

The paper identifies a novel form of cultural bias in Large Language Models (LLMs) termed Insider-Outsider Bias.

• The Phenomenon: LLMs tend to adopt an "insider" perspective (demonstrating deep familiarity, shared values, and nuanced understanding) when generating content about mainstream, dominant cultures (specifically the United States). Conversely, they default to an "outsider" perspective (treating the culture as exotic, traditional, static, or requiring explanation) when generating content about non-Western or less dominant cultures.
• The Consequence: This leads to Representational Harm (over-representing Western subjective values) and Allocational Harm (allocating resources/attention differently based on cultural standing). It reinforces Eurocentric and Americentric worldviews by framing non-Western cultures as "subjects of external observation" rather than equal participants in a dialogue.
• Research Gap: Previous work focused on explicit stereotypes or value alignment with assigned personas. This paper investigates the default cultural stance LLMs adopt in open-ended generation tasks without explicit cultural identity assignment.

2. Methodology

A. The INSIDEOUT Benchmark

To quantify this bias, the authors constructed the INSIDEOUT benchmark:

• Task: Interview Script Generation. The LLM acts as a journalist interviewing a local person.
• Scale: 4,000 generation prompts.
• Cultural Scope: 10 diverse cultures across 5 continents (USA, China, Russia, Zambia, Papua New Guinea, Mexico, India, UAE, Pakistan, Cuba).
• Variables: Prompts vary by interviewee demographics (age, gender, name, occupation) to ensure generalizability.
• Evaluation Mechanism: An automated LLM-as-a-Judge (specifically gpt-o4-mini) classifies each generated script as either an "insider" or "outsider" stance based on linguistic cues, assumptions of shared knowledge, and framing.

B. Evaluation Metrics

Three metrics were proposed to quantify the bias:

1. Cultural Externality Percentage (CEP): The percentage of scripts where the LLM adopts an "outsider" stance for a specific culture.
2. Cultural Perspective Deviation (CPD): The standard deviation of CEP scores across all 10 cultures. A high CPD indicates inconsistent positioning (insider for some, outsider for others).
3. Cultural Alignment Gap (CAG): The difference in average CEP between a control group (USA) and a reference group (non-US cultures). A high CAG indicates a systematic bias favoring the US.

C. Mitigation Strategies

The paper proposes and compares two inference-time mitigation approaches:

1. Fairness Intervention Pillars (FIP): A prompt-based baseline. It injects explicit, task-specific guidelines (e.g., "Avoid assumptions," "Use open-ended prompts," "Avoid exoticism") directly into the generation prompt.
2. Mitigation via Fairness Agents (MFA): An agentic framework using multi-step reasoning and refinement. It includes three variants:
   • MFA-SA (Single-Agent): The LLM performs a self-reflection and refinement loop to critique and rewrite its own script based on fairness pillars.
   • MFA-HA (Hierarchical-Agent): Uses two specialized agents. A Critique Agent evaluates the initial script against fairness guidelines, and a Refinement Agent rewrites the script based on that feedback.
   • MFA-Plan (Agentic Planning): An autonomous pipeline with an Orchestrator that manages an iterative loop. It diagnoses issues, creates a structured change plan, executes rewrites, and validates the result until fairness criteria are met or a max iteration limit is reached.

3. Key Contributions

1. Benchmarking Insider-Outsider Bias: Introduction of the INSIDEOUT benchmark and the first systematic quantification of the "insider vs. outsider" stance in LLMs across diverse cultures.
2. Empirical Discovery: Demonstration that state-of-the-art LLMs (ChatGPT, Llama, Mistral, DeepSeek, Qwen) overwhelmingly adopt an insider stance for the US (>88% insider tone) while defaulting to an outsider stance for non-Western cultures (often >60% outsider tone).
3. Agentic Mitigation Framework: Proposing the MFA framework, which proves that structured, multi-agent workflows are significantly more effective at mitigating subtle cultural biases than static prompting alone.

4. Experimental Results

Baseline Findings (Bias Existence)

• Systematic Bias: All 5 evaluated models showed a consistent pattern of Americentrism.
  • Example: GPT-4o showed only 6.50% outsider stance for the US, but 72.53% for Papua New Guinea.
  • CPD & CAG: High deviation scores confirmed that models treat cultures unequally.
• Qualitative Analysis: Lexical analysis revealed that US scripts used ideologically rich terms (e.g., "individualism," "inclusion"), while non-US scripts relied on stereotypes and traditional tropes (e.g., "tribes," "sacrifice," "traditional values").

Mitigation Performance

The study evaluated FIP and the three MFA variants on the same 5 models:

• FIP (Prompt-based): Showed improvement but was limited. It often failed to fully correct the deep-seated bias.
• MFA-SA (Single-Agent): Achieved significant reductions in bias.
  • Average Reduction: 53.81% reduction in CPD and 60.84% in CAG.
• MFA-HA (Hierarchical-Agent): Outperformed SA, achieving:
  • 59.70% reduction in CPD.
  • 62.52% reduction in CAG.
  • Specific Success: Reduced Llama's bias by 89.70% (CAG) and Qwen's by 82.54% (CAG).
• MFA-Plan (Agentic Planning): Achieved the best results on ChatGPT, yielding a 77.00% reduction in CPD and 77.78% reduction in CAG.

Key Insight: The agentic approaches (MFA) consistently outperformed the prompt-based baseline (FIP), demonstrating that iterative critique and structured planning are necessary to overcome the models' default cultural heuristics.

5. Significance and Implications

• Beyond Stereotypes: The paper shifts the focus from detecting explicit hate speech or stereotypes to identifying structural positioning bias—how models implicitly frame their relationship to different cultures.
• Agentic AI for Fairness: The results suggest that agent-based workflows (critique, planning, refinement) are a promising direction for mitigating complex social biases in generative AI, offering a more robust solution than simple prompt engineering or fine-tuning.
• Cultural Hegemony: The findings highlight the risk of LLMs reinforcing Western cultural hegemony by treating Western culture as the "default" insider perspective and all others as "exotic" outsiders.
• Practical Application: The proposed MFA framework provides a deployable, inference-time solution for applications like journalism, education, and cross-cultural communication where neutral, respectful representation is critical.

6. Limitations

• Dataset Scope: The study covers 10 countries; future work needs to expand to more underrepresented cultures.
• Multilingual Output: The study focused on English outputs; multilingual bias was not systematically analyzed.
• Resource Constraints: Experiments were limited by computational costs, preventing evaluation on larger model scales or more extensive datasets.