💬 NLP

Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

This paper introduces LocQA, a multilingual benchmark revealing that large language models exhibit implicit global biases toward US-centric answers and intra-lingual biases favoring high-population locales, with these structural biases being exacerbated by instruction tuning.

Original authors: Guy Mor-Lan, Omer Goldman, Matan Eyal, Adi Mayrav Gilady, Sivan Eiger, Idan Szpektor, Avinatan Hassidim, Yossi Matias, Reut Tsarfaty

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guy Mor-Lan, Omer Goldman, Matan Eyal, Adi Mayrav Gilady, Sivan Eiger, Idan Szpektor, Avinatan Hassidim, Yossi Matias, Reut Tsarfaty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, multilingual librarian named "LLM" (Large Language Model). This librarian has read almost every book in the world and can speak 12 different languages fluently. You might think, "Great! If I ask this librarian a question in French, they will know exactly what life is like in France. If I ask in Spanish, they'll know about Mexico or Spain."

But this paper, "Location Not Found," reveals a surprising problem: The librarian is fluent in the language, but they are often confused about where the person asking is actually standing.

Here is the breakdown of what the researchers found, using some simple analogies.

1. The "Ambiguous Question" Test

The researchers created a game called LocQA. Instead of asking, "What is the capital of France?" (which is easy), they asked questions that sound the same but have different answers depending on where you are.

The Question: "What is the emergency phone number?"
The Trap: If you ask this in English, the answer could be 911 (USA), 999 (UK), or 000 (Australia). The question doesn't say which country you are in.

The researchers wanted to see: When the librarian doesn't know where you are, which answer do they guess?

2. The Two Big Biases (The "Glitches")

The study found that the librarian has two main habits that cause them to get it wrong.

Bias A: The "American Default" (Global Bias)

The Metaphor: Imagine a map where the United States is painted in bright neon pink, and every other country is faded gray. Even when you ask the librarian a question in Indonesian or German, they keep looking at the neon pink map.

What happens: Even when asked in a non-English language, the model often gives the American answer.
Example: If you ask in French, "When does the tax year start?" the model might say "January 1st" (the US/UK standard) instead of the specific date for France or Canada, even though French is spoken in many places.
The Finding: The model treats the US as the "standard setting" for the whole world, like a video game that defaults to "New York" even if you selected "Paris" as your character's home.

Bias B: The "Popularity Contest" (Regional Bias)

The Metaphor: Imagine a room full of people speaking Spanish. There are 20 different countries represented. The librarian acts like a popularity contest judge. They ignore the quiet kid in the corner (a small country like Bolivia) and only listen to the loud, popular kids (big countries like Mexico, Spain, and the USA).

What happens: When a language is spoken in many places (like Spanish or French), the model assumes the answer belongs to the country with the most people.
The Finding: The model becomes a "demographic probability engine." It erases smaller nations. If you ask about a law in Spanish, the model is likely to give you the answer for Mexico or Spain, effectively pretending that smaller Spanish-speaking countries don't exist.

3. The "Helpful" Trap (Instruction Tuning)

The researchers tested two types of librarians:

The Raw Librarian: Just reads books and answers.
The "Trained" Librarian: Has been taught how to be polite, helpful, and follow instructions (this is called "Instruction Tuning").

The Surprise: The "Trained" Librarian was actually worse at being culturally neutral!

Why? The training taught the librarian to be "helpful" by giving more options. So, instead of just giving the local answer, the Trained Librarian says: "The answer is X, but in the US, it's Y, and in Canada, it's Z."
The Result: By trying to be inclusive and list every possibility, they accidentally made the US answer the "main character" of the story. They didn't erase the local answer, but they buried it under a mountain of US comparisons.

4. The "Cultural Alignment Tax"

The paper calls this the "Cultural Alignment Tax."
Think of it like this: To make the AI safer and more helpful for everyone, the developers "aligned" it. But the cost of this safety was cultural erasure. In trying to make the AI a "universal assistant," they accidentally turned it into a "US-centric assistant" that just happens to speak other languages fluently.

The Big Takeaway

The paper argues that speaking a language fluently is not the same as understanding the culture.

Just because an AI can write a perfect poem in Hindi doesn't mean it knows what life is actually like in India versus the US. The researchers are calling for a new way of building AI: one that treats location as a specific, important setting (like a "region" button on a video game), rather than assuming that speaking a language automatically means you know the local rules.

In short: The AI is a polyglot (speaks many languages), but it's a tourist who always assumes they are in America, even when they are standing in Tokyo, Paris, or Buenos Aires.

1. Problem Statement

Multilingual Large Language Models (LLMs) have largely closed the gap in linguistic fluency across languages. However, this advancement masks a critical deficiency: localization. While models can generate fluent text in a target language, they often fail to ground that text in the correct regional reality (e.g., laws, measurements, cultural norms) associated with that language.

The core problem is locale-ambiguity. Many natural language queries (e.g., "What is the emergency phone number?" or "When does the fiscal year end?") are inherently ambiguous because the answer depends on the specific country or region, not just the language. Current evaluations often conflate fluency with localization or rely on explicit prompting (e.g., "What is the law in France?"), which acts as an oracle and hides the model's implicit biases. The authors argue that when models resolve these ambiguities without explicit context, they reveal their latent priors, often defaulting to a US-centric worldview or prioritizing demographics over geographic fairness.

2. Methodology: The LocQA Benchmark

To quantify these biases, the authors introduce LocQA, a diagnostic benchmark designed to test implicit priors.

Dataset Construction:
- Scope: 2,156 questions across 12 languages (English, Spanish, French, German, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Chinese) covering 49 distinct locales.
- Design: Questions are semantically invariant but locale-dependent. They contain no explicit indicators of the target region other than the language itself.
- Ground Truth: Annotators provided locale-specific answers for each question. If a question has no answer in a specific locale (e.g., "Who was the first president?" in a monarchy), it is marked N/A.
- Validation: A two-stage cross-validation process involving bilingual annotators and manual review ensured data quality.
Evaluation Metrics:
The authors define two primary metrics to measure bias:
1. Global Bias ( $B_{US}$ ): Measures the extent to which models default to US norms even when queried in non-English languages. It calculates the difference between the observed frequency of US answers in model outputs and the expected frequency based on data collisions (where US answers happen to match other locales).
  - Formula: $B_{US} = P_{obs}(A_{US}) - P_{exp}(A_{US})$
2. Regional Bias ( $B_R$ ): Measures intra-lingual bias. It assesses whether models over-represent or under-represent specific locales within a shared language (e.g., favoring Spain over Mexico in Spanish queries). It is calculated as a "lift" comparing the observed probability of a locale's answer against its expected probability in the dataset.
  - Formula: $B_R(c) = \frac{P_{obs}(c)}{P_{exp}(c)}$
Evaluation Pipeline:
- Models: 32 models evaluated (including base and instruction-tuned variants of Gemma, Qwen, OLMo, Falcon, Granite, GPT, Claude, Gemini, etc.).
- Setup: Zero-shot generation (no explicit locale context).
- Automated Judge: A 2-stage pipeline using an LLM (Gemini-2.5-Flash) to extract answers, detect semantic collisions, and identify "Framing Style" (e.g., using the US as a conceptual anchor).

3. Key Contributions

LocQA Benchmark: A validated suite of 2,156 locale-ambiguous questions designed to isolate LLMs' default priors without explicit context.
Dual-Metric Framework: A novel methodology to quantify biases along two axes: Global Bias (US-centrism) and Regional Bias (demographic skew).
Empirical Evidence of Alignment Bias: The first large-scale demonstration that instruction tuning (alignment) exacerbates US-centric bias while simultaneously reducing regional distortion, revealing a "Cultural Alignment Tax."

4. Key Results

A. Global Bias (US-Centrism)

Prevalence: Almost all 32 models exhibit a clear US bias. The average $B_{US}$ across models is 0.24, meaning US answers appear in model outputs 50% of the time, despite only being the ground truth for ~26% of the data (due to collisions).
Bias Types: The authors categorized US-centric errors into five types:
1. Intrusion (17.5%): Gratuitously inserting US answers alongside correct local ones.
2. Erasure (6.5%): Completely replacing local reality with US norms.
3. Selection (8.7%): Preferring the US option in ambiguous contexts.
4. Framing (1.4%): Explicitly contrasting local answers with the US.
5. Valid Overlap: Coincidental correctness.
Instruction Tuning: Instruction-tuned models show significantly higher Global Bias than their base counterparts. As models become more capable, they increasingly default to US-centric views.

B. Regional Bias (Demographic Skew)

Population Correlation: Models act as "demographic probability engines." There is a strong logarithmic correlation ( $R^2 = 0.41$ ) between a locale's population size and its representation score.
Inequality: Large population centers (USA, Brazil, Mexico) and Western nations (Spain, France) are consistently over-represented. Peripheral locales (e.g., Bolivia, Haiti, Mali) suffer from systematic erasure.
Instruction Tuning Effect: Conversely, instruction tuning reduces Regional Bias. By listing multiple valid options to appear "helpful" and "inclusive," aligned models flatten the representation across locales, moving $B_R$ closer to neutrality.

C. The "Cultural Alignment Tax"

The paper identifies a trade-off:

Base Models: More likely to commit to a single local answer (lower Global Bias in some cases, but higher Regional Bias/erasure of small nations).
Instruction-Tuned Models: Attempt to be "diverse" by listing multiple options. This reduces the erasure of small nations (lower Regional Bias) but introduces the US as a universal anchor for comparison, significantly increasing Global Bias.
Conclusion: Alignment practices sacrifice cultural nuance for a "safe," generic homogeneity that defaults to US norms.

D. Domain-Specific Findings

State & Country / Language: Highest US-centricity ( $B_{US} \approx 0.23-0.30$ ) but lower regional distortion.
Leisure & Culture: Lowest US-centricity ( $B_{US} \approx 0.07$ ) but extreme Regional Bias ( $|B_R - 1| = 0.69$ ), heavily favoring populous nations.

E. Explicit Prompting

Even when explicitly told "Locale: Mexico," high-accuracy models still hallucinate US answers in roughly 6–20% of errors. The correlation between model capability and US hallucination is positive ( $r=0.64$ for high-performing models), suggesting US bias is a persistent residual failure mode, not just a lack of knowledge.

5. Significance and Implications

Redefining Fluency: The paper argues that linguistic fluency is an insufficient proxy for cultural grounding. A model can speak perfect French but still default to French (France) norms over Haitian or Congolese norms, or worse, default to US norms.
The Danger of "Safety" Alignment: Current alignment practices (RLHF/Instruction Tuning) aimed at making models "helpful" and "safe" may inadvertently enforce a US-centric worldview, erasing the specific realities of non-dominant cultures.
Future Directions: The authors call for a shift from Multilingual Modeling to Multicultural and Multi-Regional Modeling. They advocate for treating "locale" as a distinct facet that must be explicitly accounted for in training and evaluation to ensure factual adequacy for all global users.
Limitations: The study covers 12 languages and 49 locales (a fraction of global diversity), relies on automated judges, and focuses on factual localization rather than subjective moral values.

In summary, LocQA exposes that current multilingual LLMs are not neutral global observers but are structurally biased toward US norms and population size, a bias that is paradoxically reinforced by the very alignment techniques intended to make them more helpful.