💬 NLP

Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA

This paper introduces a novel QA dataset to demonstrate that Large Language Models struggle to answer questions about knowledge unique to local language varieties (Cantonese and Bavarian) compared to their standard counterparts (Mandarin and German), highlighting significant information asymmetries and the critical need for improved cultural inclusivity in AI.

Original authors: Renhao Pei, Siyao Peng, Verena Blaschke, Robert Litschko, Barbara Plank

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Renhao Pei, Siyao Peng, Verena Blaschke, Robert Litschko, Barbara Plank

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, all-knowing librarian named LLM (Large Language Model). This librarian has read almost every book in the world, but there's a catch: they mostly read the "Standard Edition" of books. They know everything about the "Official Version" of a topic, but they often miss the colorful, local details found in the "Regional Edition."

This paper is like a detective story where the authors ask: "What happens when you ask our librarian about a local secret that only exists in the Regional Edition?"

Here is the breakdown of their investigation using simple analogies.

1. The Problem: The "Local Secret" Gap

Think of Wikipedia as a massive library with different sections for different languages.

The Standard Section: This is like the main library in the city center. It's huge, well-organized, and has millions of books (e.g., Mandarin Chinese, Standard German).
The Local Section: This is like a cozy, neighborhood library. It has fewer books, but it holds local secrets that the big city library doesn't know about.
- Example: A specific folk dance in Bavaria might have a detailed history in the Bavarian dialect Wikipedia, but the Standard German Wikipedia page just says, "It's a dance."
- Example: A specific type of ancient Chinese opera might be described in Cantonese Wikipedia with details about its origins, while the Mandarin version skips those details.

The authors call this "Information Asymmetry." It's like the librarian knows the general story but missed the juicy local gossip.

2. The Experiment: The "Quiz Show"

The researchers built a special quiz show called WILOVA-QA.

The Players: They tested several super-smart AI librarians (like Llama, Qwen, and GPT).
The Questions: They asked questions based only on the local secrets found in the neighborhood library (Cantonese and Bavarian pages) that were missing from the city library (Mandarin and German pages).

The Results (Round 1: Closed Book):
When they asked the AI, "Tell me this local secret" without giving them any help, the AI failed miserably.

Analogy: It's like asking a chef, "What's the secret ingredient in this specific family recipe?" and the chef says, "I don't know, I've never heard of it." Even though the recipe exists in the world, the chef's internal memory didn't have it.

The Results (Round 2: Open Book):
Then, the researchers gave the AI a cheat sheet. They handed them the text from the local Wikipedia page.

Scenario A (Local Language): They gave the text in the local dialect (e.g., Bavarian).
- Result: The AI suddenly got it right! It could read the local text and answer the question.
Scenario B (Translated): They translated the local text into the standard language (e.g., translating Bavarian to German) and gave it to the AI.
- Result: The AI did even better. It seems the AI is like a student who understands the lesson best when it's explained in the language they are most comfortable with, even if the original source was in a dialect.

3. The Big Discovery: The "Missing Pages"

The study found that these "local secrets" aren't just rare; they are systematically missing from the AI's brain.

If a fact is only in the local Wikipedia, the AI doesn't know it exists.
If you give the AI the local text, it can reason through it perfectly.
The Takeaway: The AI isn't "dumb"; it just hasn't been fed the "Regional Edition" of the encyclopedia during its training. It's like a student who studied the textbook but never read the local newspaper.

4. Why This Matters

The authors argue that this is a problem of inclusivity.

If we only train AI on "Standard" languages, we lose the unique cultural flavor, history, and local knowledge of millions of people.
The AI becomes a "Global Generalist" that knows a little bit about everything, but misses the deep, specific details that make a culture unique.

The Bottom Line

Imagine the AI as a tour guide.

Right now, if you ask the guide about a famous landmark, they give a perfect, standard tour.
But if you ask about a hidden, local street festival that only the locals know about, the guide says, "I don't know."
The Solution: If you hand the guide a local map (the context), they can instantly tell you everything about the festival.

The paper's conclusion: To make AI truly smart and culturally aware, we need to stop ignoring the "neighborhood libraries" and start feeding the AI the local, dialect, and regional stories that are currently missing from its memory.

1. Problem Statement

Large Language Models (LLMs) are increasingly used as primary knowledge sources, yet their reliability varies significantly across languages and dialects. A critical issue is information asymmetry: local language varieties (e.g., Cantonese, Bavarian) often contain unique factual information in their Wikipedia editions that is absent from their higher-resource standard counterparts (e.g., Mandarin, German).

Current LLMs struggle with:

Closed-book retrieval: Failing to recall facts that exist only in low-resource local varieties because they are underrepresented in pre-training data.
Cross-lingual context retrieval: Difficulty in retrieving and reasoning over information when the context is provided in a local dialect but the query is in a standard language (or vice versa).
Cultural bias: The tendency of models to rely on high-resource languages, leading to a "content gap" where regional knowledge is inaccessible to users querying in standard languages.

2. Methodology

A. Dataset Construction: WILOVA-QA

The authors introduce WILOVA-QA (Wikipedia Local Variety Question Answering), a novel dataset designed to evaluate LLMs on information asymmetry.

Target Pairs: The study focuses on two pairs of overlapping regional/standard language varieties:
1. Cantonese (yue) vs. Mandarin Chinese (cmn)
2. Bavarian (bar) vs. German (deu)
Data Filtering:
- Aligned Wikipedia pages were extracted.
- A "local-heavy" filter was applied, retaining only pairs where the local language lead section and full document were longer than the standard language version.
- Direct translations and structural mismatches (e.g., disambiguation pages) were removed.
Annotation Levels:
1. Lead Section QA: 294 pairs for Cantonese-Mandarin and 178 for Bavarian-German. Annotators identified facts present only in the local lead section and formulated questions in the standard language.
2. Document-Level QA: 80 (yue-cmn) and 46 (bar-deu) pairs where information was found in the full local article but absent from the standard article entirely.
Quality Control: Native speakers verified grammaticality, stand-alone validity, and answer appropriateness.

B. Experimental Setup

Models: Six state-of-the-art LLMs were evaluated: Llama3.1 (8B, 70B), Qwen2.5 (7B, 72B), and gpt-oss (20B, 120B).
Context Scenarios:
1. Question-only: Closed-book setting (no context).
2. +Standard: Context provided from the standard language Wikipedia lead section.
3. +Local: Context provided from the local language Wikipedia lead section.
4. +Standard+Local: Both contexts provided.
5. +Local (Translated): Local context translated into the standard language (applied only to Cantonese-Mandarin due to lack of high-quality Bavarian-German MT).
Baselines: The study also evaluated on ECLeKTic, a cross-lingual dataset, to compare performance on standard vs. local varieties.
Metrics: ROUGE-L, chrF++, BERTScore, and LLM-as-a-Judge (LLMaJ) for accuracy verification.

3. Key Contributions

WILOVA-QA Dataset: The first benchmark specifically targeting information asymmetry between local dialects and standard languages, highlighting facts unique to low-resource editions.
Empirical Evidence of Asymmetry: Demonstrated that LLMs systematically fail to recall local knowledge in closed-book settings, confirming that training data gaps mirror web-based information asymmetries.
Contextual and Translational Insights:
- Provided local context significantly boosts performance, proving LLMs can reason over this knowledge if explicitly supplied.
- Translating local context to the standard language (e.g., Cantonese $\to$ Mandarin) further improves performance, suggesting that context language alignment aids knowledge integration.
Granular Analysis: Stratified evaluation by topic (e.g., geography, entertainment) and region (local vs. global) revealed that local-relevant articles are not inherently harder for models than global ones, provided the correct context is available.

4. Key Results

Closed-Book Performance

LLMs consistently failed to answer questions derived from local Wikipedia lead sections without context.
Example: On the Bavarian-German pair, Llama3.1-70B achieved a BERTScore of ~17.9 in the question-only setting, indicating near-random performance on local facts.

Open-Book Performance (Context Provided)

+Local Context: Providing the local language lead section resulted in massive performance gains.
- Cantonese-Mandarin: BERTScore increased from ~17.0 to ~54.0 (Qwen2.5-72B).
- Bavarian-German: BERTScore increased from ~17.9 to ~31.1 (Qwen2.5-72B).
Translation Effect: For Cantonese, translating the local context to Mandarin (+local translated) yielded the highest ROUGE-L and chrF++ scores, though LLM-as-a-Judge accuracy remained highest with the original local context for some models.
Standard Context: Providing the standard language context (+standard) often yielded marginal gains or even degraded performance compared to the baseline, as the standard page lacked the specific answer.

Cross-Lingual Comparison (ECLeKTic)

Performance on WILOVA-QA (local varieties) was significantly lower than on ECLeKTic (standard high-resource languages), confirming that low-resource dialects pose a greater challenge for cross-lingual retrieval.

Topic and Region Analysis

Geography and Entertainment were the most frequent topics.
Contrary to expectations, local-region articles were not more difficult for models to answer than "other" (global) articles when the correct local context was provided. This suggests the bottleneck is the absence of information in the standard training data, not an inability to process regional concepts.

5. Significance and Implications

Inclusivity in AI: The study highlights a critical inclusivity gap. LLMs currently function as "standard-language" knowledge bases, rendering unique regional knowledge inaccessible unless explicitly prompted with local dialects.
Retrieval-Augmented Generation (RAG): The findings suggest that RAG systems for multilingual applications must prioritize retrieving from local/dialectal sources rather than relying solely on standard language corpora to ensure factual completeness.
Cultural Representation: The "content gap" in Wikipedia directly translates to a "knowledge gap" in LLMs. To build truly inclusive AI, training data and retrieval strategies must be expanded to include diverse language varieties.
Future Work: The authors call for better machine translation tools for low-resource dialects (like Bavarian) and further research into aligning model knowledge with regional cultural contexts.

In summary, the paper provides robust evidence that while LLMs possess the reasoning capability to handle local dialect knowledge, they lack the parametric memory to recall it without external, dialect-specific context, thereby perpetuating information asymmetry between global and local communities.