💬 NLP

Do Chinese models speak Chinese languages?

This study reveals that despite China's diverse linguistic landscape, its leading open-weight LLMs exhibit multilingual performance profiles strikingly similar to Western models—excelling in major global languages like Mandarin, French, and German while often failing to support minority languages like Kazakh and Uyghur—suggesting that global benchmarking practices and shared training resources are driving a homogenization of language capabilities over localized prioritization.

Original authors: Andrea W Wen-Yi, Unso Eun Seo Jo, David Mimno

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrea W Wen-Yi, Unso Eun Seo Jo, David Mimno

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence as a massive, global cooking competition. Chefs (model developers) from different countries are trying to create the ultimate "Universal Language Pot" that can understand and speak every language on Earth.

This paper asks a specific question: Do the chefs from China, who have a kitchen full of diverse local ingredients, actually cook with them? Or do they just follow the same recipe as the chefs from the US and Europe?

Here is the breakdown of their findings, using simple analogies:

1. The Setup: A Diverse Kitchen vs. A Global Menu

China is like a massive kitchen with 1.4 billion people speaking Mandarin, plus hundreds of other local dialects and minority languages (like Uyghur, Tibetan, and Kazakh). You would expect the Chinese chefs to have a special advantage: they have easy access to these local ingredients.

However, there is a catch. To win the competition and get famous globally, the chefs are judged by a panel of critics who mostly speak English and use English-based scorecards (benchmarks). If you want to be recognized as a top chef, you have to make your dish taste great to these English-speaking judges.

2. The Experiment: Tasting the Dishes

The researchers tasted dishes from 6 Chinese chefs (like Qwen, Yi, and DeepSeek) and 4 Western chefs (like Llama and Mistral). They tested them on 21 different "languages," ranging from major ones like French and German to local Chinese minority languages.

They used three different ways to taste the food:

The Translation Test: Can the model translate a sentence from English to another language without losing the flavor (meaning)?
The Reading Test: Can the model read a story in a foreign language and answer questions about it correctly?
The Name Game: Can the model look at a text and correctly say, "This is written in Kazakh" or "This is written in Tibetan"?

3. The Results: The "Mandarin Exception"

The results were surprisingly uniform, with one big exception.

The "Clone" Effect: For almost every language tested (French, German, Japanese, Korean, etc.), the Chinese models and the Western models performed almost identically. It's as if both groups of chefs were following the exact same global recipe book. The paper found a 93% correlation between them. If a Western model was good at French, the Chinese model was also good at French. If a Western model struggled with a minority language, the Chinese model struggled just as much.
The Mandarin Superpower: The only time the Chinese chefs truly outshined the Western ones was with Mandarin Chinese. Because they had more access to Mandarin data and focused on it, their "Mandarin dish" was significantly tastier than the Western chefs' version.
The Minority Language Blind Spot: Here is the sad part. Despite having access to local ingredients, the Chinese models were just as bad at handling Chinese minority languages (like Uyghur and Kazakh) as the Western models were. In fact, many Chinese models couldn't even recognize that a text was written in Uyghur or Kazakh. They treated these local languages as if they didn't exist, just like the Western models did.

4. Why Did This Happen?

The paper suggests two main reasons:

The "Global Scorecard" Pressure: To get famous and get funding, Chinese developers are obsessed with winning global benchmarks. These benchmarks are mostly in English or major European languages. So, instead of using their unique access to local Chinese data to build a truly diverse model, they optimized their models to look good on the global scoreboard.
The "Mandarin-First" Policy: While China has a history of trying to protect its many languages, recent policies have focused heavily on making Mandarin the single, unifying language for the whole country. The AI developers seem to be following this trend: they prioritize Mandarin (the "super language") and ignore the rest.

The Bottom Line

The paper concludes that despite China having a unique linguistic landscape and the potential to build a model that truly speaks to its diverse population, the Chinese models have become "homogenized."

They are essentially carbon copies of Western models, with one small upgrade: they speak Mandarin slightly better. They have not used their unique position to save or elevate the hundreds of other languages spoken within China's borders. Instead, they are all cooking the same global dish, leaving the local, minority flavors on the shelf.

Technical Summary: Do Chinese models speak Chinese languages?

Problem Statement
The rapid development of Large Language Models (LLMs) in China has established the country as a leading force in AI. However, a critical tension exists for Chinese model developers: they must navigate the trade-off between serving a linguistically diverse domestic population (speaking Mandarin, hundreds of dialects, and minority languages) and optimizing for globally visible benchmarks that are predominantly English. While China has a history of language policies ranging from pluralism to assimilation, and recent AI regulations emphasize data diversity, it remains unclear whether Chinese-developed open-weight LLMs prioritize local linguistic diversity or align with global benchmarking pressures. Specifically, it is unknown if Chinese models possess superior capabilities in Mandarin and Chinese minority languages compared to Western-developed models, or if they exhibit a homogenized multilingual performance profile similar to their Western counterparts.

Methodology
The authors conducted a comparative study of 10 open-weight LLMs: 6 developed by Chinese organizations (Qwen2.5, Yi-1.5, DeepSeek-R1 variants, InternLM3, Baichuan2) and 4 Western-developed models (Llama-3, Mistral, OLMo-2, Gemma-2). All models were restricted to the 7–9 billion parameter scale to ensure fair comparison. The study evaluated multilingual performance across 21 language variants, categorized into Mandarin Chinese, Chinese Han dialects, Chinese minority languages (Jingpho, Lhasa Tibetan, Uyghur, Mongolian, Kazakh), Northeast Asian, Southeast Asian, and European languages.

The evaluation utilized three distinct experiments:

Information Parity (IP): Using the FLORES+ machine translation benchmark, the authors measured the efficiency of representing information in target languages relative to English. IP is calculated as the ratio of the negative log-likelihood (NLL) of the reference text to the target text.
Machine Reading Comprehension (MRC): Using the Belebele dataset, the authors evaluated zero-shot multiple-choice reading comprehension accuracy across 17 languages.
Language Identification: Using the Multilingual Corpus of Minority Languages in China (MC2), the authors tested the models' ability to identify four specific Chinese minority languages (Tibetan, Mongolian, Kazakh, Uyghur) from text samples.

Key Contributions

Hypothesis Testing: The paper proposes and tests four hypotheses regarding Chinese developers' priorities: the Null Hypothesis (global benchmark optimization leads to parity with Western models), the Mandarin Hypothesis (prioritization of Mandarin only), the Pluralist Hypothesis (broad support for all local languages), and the Regional Hypothesis (support for major Asian languages).
Empirical Benchmarking: The study provides a granular, quantitative comparison of Chinese and Western LLMs across a diverse set of languages, including under-represented minority languages often omitted in standard benchmarks.
Policy Analysis: The work connects technical performance metrics with China's historical and current language policies, analyzing how political and economic incentives shape AI development priorities.

Results

High Correlation: The multilingual performance of Chinese-developed and Western-developed models is highly correlated ( $r \approx 0.93$ for IP and $r \approx 0.99$ for MRC). The performance profiles are nearly identical across most language categories, suggesting that Chinese models do not significantly outperform Western models in non-Mandarin local languages.
Mandarin Superiority: The sole exception to this homogenization is Mandarin Chinese. Chinese-developed models consistently outperform their Western counterparts in both Information Parity and Machine Reading Comprehension for Mandarin (Simplified and Traditional).
Minority Language Deficits: Contrary to the Pluralist Hypothesis, Chinese models perform poorly on Chinese minority languages, often failing to identify languages such as Kazakh and Uyghur. Their performance on these languages is comparable to, or sometimes worse than, Western models.
Regional Performance: Chinese models perform well on European languages (French, German) and major Asian languages (Japanese, Korean), mirroring the strengths of Western models rather than showing unique regional advantages.
Instruction Tuning: Instruction-tuned models generally outperform base models, but the performance gap between Chinese and Western models remains consistent regardless of tuning.

Significance and Claims
The paper argues that the observed homogenization in multilingual performance indicates that Chinese model developers are primarily optimizing for global benchmarking practices and shared training resources rather than addressing the specific linguistic needs of China's diverse domestic population. The results support the Mandarin Hypothesis (Chinese models are better at Mandarin but not other local languages) and fail to reject the Null Hypothesis (high correlation with Western models).

The authors interpret these findings as evidence that global benchmarking pressures and shared data resources exert a "normalizing pressure" on LLM training, leading to similar performance profiles across distinct political and cultural contexts. This suggests that current language support in LLMs is a result of prioritization and trade-offs rather than an inevitable outcome of development. The paper highlights the implications for policymakers and developers, noting that without specific intervention, the "sovereign AI" movement may result in environmental costs and cultural homogenization without delivering unique value for local linguistic diversity. The authors call for future work to investigate the structural and institutional factors driving these priorities and to explore opportunities for developers to leverage unique access to local data to differentiate their models.