💬 NLP

Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate

This preliminary study refutes the claim that Chinese prompts are more token-efficient than English for LLM coding tasks, revealing instead that Chinese prompts often result in lower success rates and model-dependent token costs that do not yield the anticipated savings.

Original authors: Simiao Ren (Tsang), Xingyu Shen (Tsang), Yuchen Zhou (Tsang), Dennis (Tsang), Ng, Ankit Raj

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simiao Ren (Tsang), Xingyu Shen (Tsang), Yuchen Zhou (Tsang), Dennis (Tsang), Ng, Ankit Raj

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🚫 The Big Myth: "Speak Chinese to Save Money"

Imagine you're running a delivery service (that's your AI coding assistant). You have a popular rumor going around the neighborhood: "If you write your delivery instructions in Chinese, the trucks will be smaller, carry more packages, and cost you 40% less in gas."

Because of this rumor, many developers started switching their coding prompts from English to Chinese to save money on API fees. They thought Chinese characters were like "super-dense" bricks that could build a wall with fewer pieces than English bricks.

This paper is the "MythBusters" team that went out to test that rumor. They built a lab, ran the trucks, and measured the gas.

The Verdict? The rumor is false. In fact, speaking Chinese might actually make your delivery service more expensive and less reliable.

🧪 How They Tested It (The Experiment)

The researchers didn't just guess; they set up a rigorous race.

The Track (The Task): They used a famous obstacle course called SWE-bench. Imagine 50 different broken machines (bugs in software code). The AI's job is to read the manual, find the broken part, and fix it so the machine works again.
The Drivers (The Models): They used three different types of AI drivers:
- MiniMax-2.7: A Chinese-developed driver.
- GLM-5: Another Chinese-developed driver.
- GPT-5.4-mini: An English-developed driver.
The Instructions: They gave the exact same broken machine instructions to each driver, but half the time in English and half the time in Chinese.
The Scorecard: They counted two things:
- Did it work? (Did the AI fix the bug?)
- How much did it cost? (How many "tokens" or digital words did the AI use to solve it?)

📉 The Three Big Surprises

Here is what they found, broken down simply:

1. The "Chinese is Cheaper" Myth is Dead

The rumor said Chinese prompts use fewer tokens. The data said: "Not so fast."

The MiniMax Driver: When speaking Chinese, it actually used 28% more tokens (more gas) than when speaking English.
The GLM Driver: It used slightly fewer tokens in Chinese, but not enough to matter.
The GPT Driver: It used a bit more in Chinese.

The Analogy: Think of tokenizers (the AI's internal dictionary) like a translator. If the translator's dictionary is built mostly for English, a Chinese word might get chopped up into three tiny pieces, while an English word stays as one big piece. So, even though the Chinese sentence looks shorter on paper, the AI has to process more pieces to understand it.

2. The "Success Rate" Trap (The Most Important Part)

This is the biggest lesson. Even if a language uses fewer tokens per attempt, it doesn't matter if the AI fails more often.

The Scenario: Imagine you are trying to open a locked door.
- English Key: Costs $1 to make. You succeed 66% of the time.
- Chinese Key: Costs $0.90 to make. But you only succeed 55% of the time.
The Result: Because the Chinese key fails more often, you have to try again and again. Those repeated failures add up.
The Math: The researchers calculated the "Cost per Successful Fix." Because the Chinese prompts failed more often across all models, the total cost ended up being higher, not lower.

The Analogy: It's like buying a cheap pair of shoes that fall apart after one mile versus a slightly more expensive pair that lasts a year. The cheap shoes look like a bargain until you realize you have to buy a new pair every week.

3. The Driver Matters More Than the Language

The biggest difference in performance wasn't between English and Chinese; it was between the AI models themselves.

The best model (MiniMax) fixed about 66% of the bugs.
The worst model (GPT) fixed only 36% of the bugs.
That 30% gap is massive. Switching languages only changed the success rate by about 5–10%.

The Analogy: It doesn't matter if you drive a Ferrari or a Toyota; if you pick the wrong driver (the AI model), you aren't going to win the race. Choosing the right AI model is 100 times more important than the language you speak to it.

💡 The "Why" Behind the Confusion

Why did everyone believe the Chinese myth? The paper explains that people mixed up three different things:

Information Density: Yes, one Chinese character can hold the meaning of an English phrase. (True!)
Tokenization: But AI doesn't read characters; it reads "tokens" (chunks of text). If the AI's dictionary isn't optimized for Chinese, those characters get chopped into tiny, expensive pieces. (The Reality!)
Reasoning: Some people thought Chinese "thinking" was shorter. But in coding, if the thinking is shorter but wrong, you have to pay to fix it again.

🏁 The Final Takeaway for Developers

If you are a developer trying to save money on AI coding:

Don't switch to Chinese just to save tokens. You probably won't save money, and you might actually spend more because the AI will fail more often.
Pick the right AI model. Spending time finding the best model (like MiniMax or GLM) will save you way more money than changing your language.
Success is the real currency. It's not about how many words you use; it's about how often the AI gets the job done on the first try.

In short: The "Vibe Coding" trend is cool, but don't let a social media rumor trick you into thinking a different language is a magic money-saving button. The engine (the model) matters more than the language you speak to it.

1. Problem Statement

A prevalent claim circulating on social media and developer forums suggests that using Chinese prompts for Large Language Model (LLM) coding tasks is significantly more token-efficient than English, potentially reducing API costs by up to 40%. This narrative is based on the intuition that Chinese characters are more information-dense than English words. Consequently, many developers are considering switching to Chinese for "vibe coding" (generating code via natural language) to save money.

However, this claim conflates character-level information density with token-level efficiency. It ignores how tokenizers (which break text into sub-word units) are trained, often favoring English due to data dominance, and fails to account for the impact of language on task resolution rates. The paper aims to rigorously test whether switching to Chinese actually yields cost savings or performance gains in software engineering tasks.

2. Methodology

The authors conducted a controlled empirical study using the SWE-bench Lite benchmark, a standard for evaluating LLMs on real-world software engineering tasks (bug fixing).

Dataset: 50 instances were selected from SWE-bench Lite (covering repositories like Django, matplotlib, pytest, etc.) using stratified random sampling.
Models Evaluated: Three distinct model families were tested to cover different architectures and tokenization strategies:
1. MiniMax-2.7: A Chinese-developed model with potential CJK optimizations.
2. GPT-5.4-mini: An English-dominant model (OpenAI) serving as a contrast.
3. GLM-5: A Chinese-developed model with explicit Chain-of-Thought (CoT) reasoning.
Experimental Design:
- Bilingual Prompts: Each task was presented in both English and professionally translated Chinese (preserving technical terminology).
- Two-Phase Execution:
  1. Patch Generation: An agent (MiniSWEAgent) attempted to generate code patches within a 1,500-step limit.
  2. Evaluation: Generated patches were applied to the codebase, and test suites were run to determine if the bug was resolved.
- Metrics: The study tracked token consumption (prompt, completion, reasoning), API costs, and resolution rates (success/failure).
- Cost Efficiency Metric: The authors introduced a critical metric: Expected Cost per Successful Task ( $C_{eff}$ ), calculated as:
  $C_{eff} = \frac{\text{Avg. Token Cost per Attempt}}{\text{Resolution Rate}}$
  This accounts for the fact that failed attempts require retries, amplifying total costs.

3. Key Contributions

Systematic Empirical Validation: The first rigorous study comparing English vs. Chinese prompts specifically for code reasoning tasks using a real-world benchmark (SWE-bench), moving beyond anecdotal evidence.
Granular Token Analysis: Distinguishing between prompt, completion, and reasoning tokens, and analyzing how different model architectures handle language-specific tokenization.
Holistic Cost-Efficiency Framework: Moving beyond raw token counts to evaluate the "Expected Cost per Successful Task," demonstrating that success rate is a dominant factor in total cost.
Tokenizer Mechanism Analysis: Isolating the role of tokenizer vocabulary allocation (e.g., English-dominant vs. CJK-optimized) in determining token efficiency.

4. Key Results

A. Resolution Rates (Success Rates)

Chinese prompts generally performed worse or equal to English.
- MiniMax-2.7: English (66.0%) vs. Chinese (61.5%).
- GPT-5.4-mini: English (36.0%) vs. Chinese (26.1%).
- GLM-5: English (64.6%) vs. Chinese (55.1%).
The performance gap between models (approx. 30 percentage points) was far larger than the gap caused by language choice.

B. Token Cost & Efficiency

No Universal Advantage: The assumption that Chinese is always more token-efficient was false.
- MiniMax-2.7: Chinese required 1.28x more tokens than English.
- GPT-5.4-mini: Chinese required 1.09x more tokens.
- GLM-5: Chinese required 0.98x tokens (slightly fewer), likely due to its CJK-optimized tokenizer.
The "Cost Penalty" of Failure: Even when Chinese used fewer tokens per attempt (GLM-5), the lower resolution rate meant the expected cost per successful task was higher for Chinese. The cost of retrying failed attempts outweighed the minor token savings.

C. Tokenizer Analysis

The study confirmed that token efficiency is model-dependent, not language-dependent.
Tokenizers trained on English-dominant data (e.g., GPT, Llama) fragment Chinese characters into multiple tokens, increasing token counts.
Tokenizers trained with significant Chinese data (e.g., GLM) can compress Chinese more efficiently, sometimes better than English.
Character Density vs. Token Efficiency: While Chinese characters are denser (more semantic info per character), this does not translate to fewer tokens unless the tokenizer is specifically optimized for CJK.

5. Significance and Implications

Debunking the Myth: The paper conclusively "busts" the myth that switching to Chinese prompts for coding will save money. In most tested scenarios, it either increased costs or provided negligible savings.
Strategic Shift for Practitioners:
- Model Selection > Language Choice: Choosing the right model architecture has a far greater impact on performance and cost (30% variance) than the language of the prompt (<10% variance).
- Focus on Success Rate: Optimizing for the highest probability of a successful patch is more cost-effective than trying to minimize input token counts.
- Tokenizer Awareness: Users relying on Chinese prompts should select models with CJK-optimized tokenizers (like GLM or MiniMax) rather than English-centric ones, but even then, success rates may drop.
Research Direction: Highlights the need for future studies to include more models and languages, and to standardize token tracking across API providers to ensure accurate cost comparisons.

Conclusion: The claim that Chinese is inherently more efficient for LLM coding is unsupported by empirical data. The "efficiency" of a language is entirely mediated by the specific model's tokenizer and training data, and the higher failure rates associated with Chinese prompts in this study often lead to higher overall operational costs.