💬 NLP

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

This paper reveals that while large language models often exhibit higher raw accuracy in English due to superior general proficiency, their access to local cultural knowledge is frequently more effective in native languages once this proficiency gap is controlled for, suggesting that lower local-language performance often masks, rather than reflects, weaker cultural understanding.

Original authors: Yang Zhang, Xiao Fei, Amr Mohamed, Sarah Almeida Carneiro, Mersin Konomi, Mingmeng Geng, Ahmed Asaad, Guokan Shang, Michalis Vazirgiannis

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Zhang, Xiao Fei, Amr Mohamed, Sarah Almeida Carneiro, Mersin Konomi, Mingmeng Geng, Ahmed Asaad, Guokan Shang, Michalis Vazirgiannis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a library of books about the history, traditions, and daily life of 13 different countries. You also have a team of 80 different "super-readers" (AI models) who can read these books and answer questions about them.

The big question this paper asks is: To get the best answers about a specific country, should you ask the super-reader in English, or in that country's local language?

The Problem: The "Fluency Fog"

Previously, researchers looked at raw scores and got confused. They saw that when you asked an AI about French culture in French, it often got the answer wrong. When asked in English, it got it right. They concluded, "Aha! The AI knows more about French culture when we speak English!"

The authors of this paper say: Wait a minute. That's like judging a chef's cooking skills by how well they can read the recipe in a foreign language. If the chef is a master of French cuisine but can't read English well, they might fail a test written in English, not because they don't know the food, but because they can't read the instructions.

The paper argues that AI models are often fluent in English but clumsy in local languages. This "fluency fog" hides the fact that the AI might actually know the local culture better when you speak to it in its own tongue.

The Solution: The "Three-Step Detective"

To solve this, the researchers built a clever testing framework using a mathematical tool called IRT (Item Response Theory). Think of IRT as a special scale that weighs the difficulty of the question against the skill of the reader, so you can compare apples to apples even if the questions are different.

They tested the AI with three types of comparisons:

The "Global" Test (Proficiency Check):
- The Setup: Ask the AI general questions (like "What is the capital of France?") in both English and French.
- The Result: The AI almost always does better in English.
- The Meaning: This measures Language Proficiency. It tells us the AI is just better at reading and understanding English. This is the "fluency fog."
The "Local" Test (Raw Performance):
- The Setup: Ask the AI specific cultural questions (like "What is the traditional dish for a specific local festival?") in both English and French.
- The Result: Mixed. Sometimes English wins, sometimes French wins.
- The Meaning: This is the messy real-world score. It's a mix of "How good is the AI at French?" + "How much French culture does it actually know?"
The "Knowledge" Test (The Magic Step):
- The Setup: The researchers took the "Local" score and subtracted the "Global" score.
- The Logic: If you remove the "Language Proficiency" part, what is left? The pure Cultural Knowledge Access.
- The Result: Boom! In almost every single case, the AI knew the local culture better when asked in the local language.

The Big Discovery: The "Masked Advantage"

The paper calls this the "Masked Advantage."

Imagine a brilliant local historian who speaks perfect English but has a slight stutter when speaking their native tongue.

If you ask them a history question in English, they answer perfectly.
If you ask them in their native tongue, they stumble over the words and give a wrong answer, even though the facts are in their head.

The paper found that for AI, the "stutter" (weak language skills) is so strong that it masks the "facts" (cultural knowledge). The AI actually holds the local cultural knowledge more accessibly in the local language, but because it struggles to form the sentences, it looks like it knows less.

Key Takeaways from the Study

English is the "Safe Bet" for general smarts: The AI is consistently better at English. If you just want to know general facts, English is the most reliable language.
Local languages hold the "Secret Knowledge": Once you account for the AI's struggle with grammar and vocabulary, the local language is actually the key that unlocks the cultural knowledge.
Bigger isn't always better (for this specific issue): Even the newest, biggest AI models show this pattern. They know the culture better in the local language, but their English fluency is just so much higher that it overshadows the local advantage.
Regionally trained models help: AI models that were specifically trained on data from a specific region (like a model trained mostly on Arabic data) show a stronger local advantage, but the "masked" effect still exists.

The Bottom Line

The paper concludes that weaker performance in a local language doesn't mean the AI lacks cultural knowledge. It usually just means the AI is bad at speaking that language. The knowledge is there, waiting to be accessed, but it's currently hidden behind a wall of language difficulty.

To get the truest picture of a culture from an AI, you shouldn't just look at the final score; you have to look past the language barrier to see what the AI actually knows.

Technical Summary: The Masked Advantage

Problem Statement

Large language models (LLMs) are increasingly deployed as multilingual systems to answer culturally grounded questions. However, a critical ambiguity remains: does querying in English or the local language provide better access to local cultural knowledge? Existing evaluations suffer from two primary limitations:

Synthetic Templates: Many benchmarks rely on parallel, human-defined templates that may not reflect how cultural knowledge naturally manifests in native data.
Conflated Metrics: Raw accuracy comparisons between languages conflate general language proficiency with language-conditioned knowledge access. A model may score higher in English simply due to superior proficiency, masking the fact that the local language might actually activate the relevant cultural knowledge more effectively.

Consequently, it is unclear whether a lower local-language score indicates a lack of cultural knowledge or merely a deficiency in linguistic proficiency.

Methodology

The authors propose a controlled framework to disentangle language proficiency from localized knowledge access using a difference-in-differences design and Item Response Theory (IRT).

1. Experimental Design

The study evaluates approximately 80 models across 13 locales (including Chinese, Arabic, French, Hindi, and Telugu). The dataset consists of three matched subsets per locale:

Culture-Agnostic (CA): High-quality parallel questions from Global-MMLU requiring no local knowledge.
Culture-Specific (CS): Native-language questions sourced from regional benchmarks (e.g., CMMLU, ArabicMMLU, KMMLU) requiring local cultural knowledge.
English Counterparts (CSEN): High-quality translations of the CS items into English, verified by native speakers for translation quality and cultural relevance.

2. The 1PL-IRT Framework

To address the issue of varying item difficulties between the CA and CS pools, the authors employ a shared One-Parameter Logistic (1PL) Item Response Theory model (equivalent to the Rasch model). This model estimates:

Item Difficulty ( $b_i$ ): A shared scalar for each logical question across all languages.
Model Ability ( $\theta_{jsk}$ ): The ability of model $j$ on subset $s$ in language $k$ .

The model assumes translation equivalence (intrinsic difficulty is constant across languages) and treats language as a facet of the examinee's latent ability rather than a shift in item difficulty. This allows for "difficulty-free" language contrasts.

3. Defining the Gaps

The framework defines three key metrics based on the estimated abilities ( $\theta$ ):

GlobalGap ( $\theta_{CA, local} - \theta_{CA, en}$ ): Estimates the general language proficiency gap. A negative value indicates the model is more proficient in English.
LocalGap ( $\theta_{CS, local} - \theta_{CS, en}$ ): Captures the combined effect of proficiency and cultural knowledge access on culture-specific questions.
KnowledgeGap ($LocalGap - GlobalGap$): Isolates language-conditioned knowledge access. A positive value indicates that the local language provides better access to cultural knowledge than English, independent of proficiency.

Key Results

1. Universal English Proficiency Advantage

Across all 13 locales and nearly every model, the GlobalGap is negative. This indicates a consistent baseline where models possess higher proficiency in English than in local languages, regardless of the language's resource level.

2. Mixed Local-Language Performance

The LocalGap is mixed. It tends to be positive (favoring the local language) only for high-resource languages (e.g., Chinese, French) or when the model originates from the same region as the language. For most other settings, the LocalGap is negative, suggesting that raw accuracy favors English queries.

3. The "Masked" Knowledge Advantage

The KnowledgeGap is positive in 98% of cells. This is the paper's central finding: once the proficiency penalty is mathematically removed, local languages consistently provide better access to local cultural knowledge.

This advantage exists even in settings where the raw LocalGap is negative.
The advantage is not driven by resource levels (unlike the proficiency gap) but is slightly amplified in models that are regionally adapted or originate from the target region.
Frontier closed-source models (e.g., GPT-5.4, Claude-Sonnet-4.6) often show a positive LocalGap directly because their proficiency penalty is low enough to no longer mask the underlying knowledge advantage.

4. Scaling and Adaptation

Model Scale: As model parameters increase, the proficiency penalty (GlobalGap) decreases, making the local-language knowledge advantage more visible in raw metrics.
Language Adaptation: Supervised Fine-Tuning (SFT) on local languages significantly improves the LocalGap (by reducing the proficiency gap) but does not uniformly increase the KnowledgeGap. This suggests SFT primarily improves linguistic alignment rather than injecting new cultural knowledge.

Significance and Claims

The paper argues that weaker local-language performance in LLMs often stems from limited proficiency rather than missing cultural knowledge. The "masked advantage" hypothesis suggests that local cultural knowledge is frequently more accessible through the local language but remains hidden by the model's inability to process that language fluently.

Implications:

Evaluation: Raw accuracy is insufficient for evaluating cultural knowledge in multilingual models. Decomposing proficiency from knowledge access is necessary to avoid false negatives regarding a model's cultural capabilities.
Deployment: Improving local-language proficiency (via better multilingual pretraining or adaptation) is a viable strategy to "unlock" existing, but hidden, cultural knowledge capabilities in models.
Benchmarks: The study validates the use of real-world, native-sourced cultural questions over synthetic templates to accurately assess localized knowledge.

The authors conclude that the local language is often the superior key to cultural knowledge, but this key is currently obscured by the lock of limited language proficiency.