💬 NLP

Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models

This paper presents a cross-lingual audit framework revealing that demographic bias in large language models used for emergency police dispatch emerges systematically during ambiguous incidents, varies significantly across gender, race, and religious appearance, and exhibits distinct asymmetries between English and Mandarin Chinese, highlighting the need for context-aware, language-specific model evaluations before real-world deployment.

Original authors: William Guey, Wei Zhang, Pierrick Bougault, Yi Wang, Bertan Ucar, Vitor D. de Moura, José O. Gomes

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: William Guey, Wei Zhang, Pierrick Bougault, Yi Wang, Bertan Ucar, Vitor D. de Moura, José O. Gomes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a traffic cop at a busy intersection. Your job is to decide how urgently to send help: do you need a full SWAT team immediately, or just a regular patrol car later, or maybe no one at all?

Now, imagine you hire a super-smart robot (an AI) to help you make these decisions. The paper you asked about is like a "stress test" for these robots. The researchers wanted to see if the robots treat people differently based on who they think the people are (their race, gender, or religion), even when the emergency call itself sounds exactly the same.

Here is the breakdown of what they did and what they found, using simple analogies:

The Experiment: The "Twin" Test

The researchers created 15 different emergency scenarios (like a fight in a park, a suspicious bag in a subway, or someone being followed home).

For each scenario, they made two "twins":

Twin A: The story is the same, but the caller mentions a specific detail about a person involved (e.g., "The suspect is wearing a turban," "The suspect is a Black man," or "The suspect is a woman").
Twin B: The story is exactly the same, but that specific detail is removed or changed to a neutral description.

They fed these stories to 11 of the world's most advanced AI robots (Large Language Models) and asked them to assign a priority level, ranging from "Do nothing" to "Send help immediately." They did this in both English and Mandarin Chinese.

The Big Findings

1. The "Foggy Window" Effect

The most important discovery is that the robots are only biased when the situation is unclear.

Clear Danger: If the story says, "There is a hostage situation with a knife," every robot, regardless of who the suspect is described as, screamed "Send help immediately!" The bias disappeared.
Foggy Danger: If the story was vague, like "A man is standing near a monument for 30 minutes," the robots started making assumptions based on the demographic details.
The Analogy: Think of it like looking through a foggy window. If a car is clearly crashing (clear danger), you see it no matter what. But if you see a blurry shape in the fog (ambiguous danger), your brain (or the robot's brain) fills in the gaps based on stereotypes. The bias only happens in the fog.

2. The "Religion vs. Race" Surprise

The researchers found that the robots reacted differently to different types of clues:

Religious Appearance: This caused the biggest swings in judgment. If a story mentioned Islamic clothing (like a turban or hijab), the robots were much more likely to change their priority level compared to a neutral story.
Gender: The robots showed a specific pattern here. If the victim was described as a woman, the robots tended to send help faster than if the victim was a man.
Race: This was the most surprising part. In the US context (English), when a caller explicitly said "Black man," the robots actually lowered the urgency in some cases. This is the opposite of what we often fear (that they would be more aggressive). It's as if the robots were trying so hard not to be racist that they accidentally became too cautious.

3. The "Language Switch" Problem

The robots didn't act the same way in English as they did in Mandarin Chinese.

Gender Bias: The bias regarding women was twice as strong in Mandarin as it was in English.
Race Bias: The bias regarding race was twice as strong in English as it was in Mandarin.
The Analogy: Imagine a person who is very polite in English but very blunt in Spanish. If you only tested them in English, you would miss their bluntness. The paper warns that testing AI in just one language gives you an incomplete picture of how biased it really is.

4. The "Inconsistent Robot"

The robots didn't always follow a simple "bad stereotype" rule.

Sometimes, mentioning a Muslim name made the robot think a situation was more dangerous (escalation).
Other times, mentioning the same thing made the robot think it was less dangerous (de-escalation).
The Takeaway: It's not just a simple "robot hates X." The robot's reaction depends on the specific mix of the story and the clue. Sometimes it overreacts; sometimes it underreacts.

Why This Matters (According to the Paper)

The paper concludes that we cannot just say "AI is biased" or "AI is fair." It depends on:

How clear the emergency is: Bias hides when the danger is obvious but pops up when the situation is confusing.
Which demographic clue is used: Religion, gender, and race trigger different reactions.
The language used: A robot might be fair in English but unfair in Chinese, or vice versa.

The authors built a "playground" (an open-source tool) so that police departments can test their own AI before buying it. They want to make sure that when these robots are used in real life, they don't accidentally send the wrong kind of help to the wrong people just because of a vague description in a 911 call.

In short: The robots are smart, but they get confused in the fog. When the situation is unclear, they start guessing based on who the people are, and they guess differently depending on the language they are speaking. We need to test them carefully before letting them drive the police cars.

Technical Summary: Auditing Demographic Bias in AI-Based Emergency Police Dispatch

Problem Statement
Large Language Models (LLMs) are increasingly integrated into high-stakes public safety systems, specifically for emergency call triage and dispatch decision support. While LLMs have been evaluated for clinical and legal applications, their demographic fairness in emergency police dispatch remains largely untested. This gap is critical because dispatch systems rely on historical call records and officer-reported classifications—data sources where racial and socioeconomic biases are empirically documented. Consequently, AI systems trained on or fine-tuned with such data risk perpetuating existing disparities at an operational scale. Unlike clinical triage, police dispatch involves real-time voice transcripts, a distinct classification scale (Police Priority Dispatch System), and the direct consequence of deploying armed personnel. Existing literature lacks a systematic framework to evaluate LLM bias in this specific context using validated operational frameworks and controlled scenario designs.

Methodology: The LLM-DispatchBias Framework
The authors introduce LLM-DispatchBias, a cross-lingual audit framework designed to isolate the effect of demographic cues on dispatch classification. The methodology employs a controlled minimal-pair design across 11 frontier LLMs, 15 scenario pairs, three demographic categories, and two languages (English and Mandarin Chinese), generating 19,800 model outputs.

Classification Scale: The study operationalizes the Police Priority Dispatch System (PPDS) as a five-level ordinal scale: OMEGA (no response) to ECHO (immediate specialized response). This anchors the evaluation to real-world operational consequences rather than subjective sentiment scoring.
Scenario Design: The framework utilizes 15 scenario pairs covering the full threat spectrum (from non-emergency inquiries to active hostage situations). Each pair consists of a "Variant A" (containing a specific demographic cue) and a "Variant B" (a neutral control or a counter-cue variant).
- Religious Appearance: Islamic dress markers (turban, hijab, kufi) or Muslim names vs. no signal.
- Gender: Male-coded vs. female-coded individuals (caller, victim, or suspect) in otherwise identical incidents.
- Race: Explicit description of a suspect as a "Black man" vs. no racial description.
Experimental Controls: To ensure robustness, the study controls for surface-form variation by randomizing "opener-closer" phrases in transcripts. Models are queried at temperature zero to eliminate stochastic noise. The evaluation spans 11 models from American, Chinese, and European development traditions.
Bias Quantification: Bias is measured as the bias delta ( $\Delta$ ), calculated as the mean PPDS score of Variant A minus Variant B. A positive delta indicates escalation (higher urgency), while a negative delta indicates de-escalation.

Key Results
The study reveals that demographic bias in LLM-based dispatch is not a fixed property but arises from the interaction between demographic signals, contextual ambiguity, and language.

Bias Concentration in Ambiguity: Systematic bias emerges primarily in scenarios where incident severity is ambiguous (e.g., loitering, abandoned bags, domestic disputes without confirmed injury). In scenarios with clear, in-progress violent events (e.g., active hostage situations, witnessed assaults), models converge on the correct high-priority classification (DELTA/ECHO) regardless of demographic cues. Bias magnitude is significantly higher in ambiguous scenarios ( $p < 5.5 \times 10^{-10}$ ).
Demographic Axis Variation: The magnitude of bias varies by category. Religious appearance produced the largest mean absolute bias ( $|\Delta| = 0.193$ $∣Δ∣ = 0.193$ ), followed by gender ($0.117$) and race ($0.082$).
- Religious Appearance: Effects were heterogeneous; some scenarios (e.g., abandoned bag) triggered escalation, while others (e.g., monument loitering) triggered de-escalation, contradicting simple stereotype-amplification theories.
- Gender: Bias was directional based on victimhood. Female-coded victims received systematically higher priority than male-coded victims.
- Race: Contrary to documented human biases, explicit racial cues often led to de-escalation (reduced urgency) in ambiguous scenarios, suggesting potential RLHF over-correction or different bias structures.
Cross-Lingual Asymmetry: Bias does not transfer consistently across languages.
- Gender bias was substantially amplified in Mandarin Chinese (approx. double the magnitude of English).
- Race bias was more pronounced in English (approx. double the magnitude of Mandarin).
- Religious bias was slightly higher in English.
- Aggregate analyses across languages obscure these per-category inversions, leading to underestimation of bias in single-language evaluations.
Model Variability: Bias magnitude varied significantly across the 11 models tested. DeepSeek and Claude exhibited the highest overall bias, while Llama-4-Maverick showed the lowest. No clear East-West split in bias patterns was observed; bias profiles were highly specific to individual models and demographic categories.

Significance and Claims
The paper claims three primary contributions:

Empirical Evidence of Context-Dependent Bias: The study demonstrates that LLM bias in dispatch is not uniform; it is a property of the input context. Bias surfaces precisely where human dispatcher discretion is required (ambiguous calls), mirroring patterns found in human dispatcher literature.
Methodological Infrastructure: The primary contribution is the LLM-DispatchBias framework itself. By anchoring the audit to the stable PPDS operational framework and using a minimal-pair design, the authors provide a scalable, model-agnostic tool. This allows deploying agencies to evaluate candidate models on jurisdiction-relevant scenarios prior to real-world adoption, shifting accountability from retrospective academic documentation to proactive procurement.
Necessity of Cross-Lingual Auditing: The findings challenge the sufficiency of single-language evaluations. The paper argues that cross-lingual evaluation is a substantive measurement requirement, not merely a robustness check, as bias direction and magnitude can invert or scale significantly depending on the language of operation.

The authors conclude that while the specific models tested will be superseded, the audit design remains a durable mechanism for ensuring fairness in AI-based dispatch systems as they enter operational use.