💬 NLP

Disentangling Language Roles in Multilingual LLM Task Execution

This paper introduces MTM-Bench, a controlled benchmark with a fully crossed design of instruction, content, and response languages, to demonstrate that performance degradation in multilingual LLMs is primarily driven by the specific role a language occupies—particularly the response language—rather than simply the number of language mismatches.

Original authors: Qishi Zhan, Minxuan Hu, Seoyeon Jang, Lei Zhao, Ziheng Chen, Man Liang, Xinyue Xiang, Jiaxin Liu, Guansu Wang, Liang He

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qishi Zhan, Minxuan Hu, Seoyeon Jang, Lei Zhao, Ziheng Chen, Man Liang, Xinyue Xiang, Jiaxin Liu, Guansu Wang, Liang He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a translator to handle a very specific, tricky job. The job isn't just about knowing three languages; it's about knowing which language to use for which part of the conversation.

Usually, when we test AI, we ask: "Can this AI speak Spanish?" or "Can it understand French?" But in the real world, the situation is often messier. A boss might give instructions in Spanish, hand over a document written in English, and demand the final summary in Chinese.

This paper, titled "Disentangling Language Roles in Multilingual LLM Task Execution," introduces a new way to test AI called MTM-Bench. Think of it as a "triplet test" that isolates these three roles to see exactly where the AI gets confused.

Here is the breakdown of their findings using simple analogies:

1. The "Three-Role" Puzzle

The researchers created a benchmark with 2,430 different scenarios using English, Spanish, and Chinese. Every scenario is a unique combination of:

The Instruction: What the boss says (e.g., "Summarize this").
The Content: The material to read (e.g., an email thread).
The Response: The language the answer must be in.

They tested 20 different AI models (like the latest versions of Claude, GPT, and others) on every possible mix of these three languages.

2. The Big Discovery: The "Output Slot" is the Bottleneck

The most surprising finding is that where the language is used matters more than how many languages are mixed.

The Analogy: Imagine a chef (the AI) who is great at reading recipes in English and French. If you ask them to cook a dish but tell them to speak the instructions in a language they don't know well, they might drop the spoon.
The Result: The AI's performance dropped significantly when the Response Language (the language it had to speak/write the answer in) was different from the others.
- If the AI had to answer in English, it did great.
- If it had to answer in Spanish or Chinese, its performance took a big hit.
- Interestingly, it didn't matter as much if the instructions or the reading material were in a different language. The AI could handle the confusion in the "input" side much better than the "output" side.

3. The "Mismatch Count" Myth

A common assumption is: "The more languages you mix up, the harder the task is."

The Paper's Claim: This isn't true.
The Analogy: Think of it like wearing mismatched socks.
- Scenario A: You wear a red sock on your left foot and a blue sock on your right (One mismatch).
- Scenario B: You wear a red sock, a blue sock, and a green hat (Three mismatches).
- The researchers found that the AI struggled just as much with Scenario A (where only the output language was wrong) as it did with Scenario B (where everything was mixed up).
- Conclusion: Once the AI has to switch languages to give the answer, adding more mixed languages to the instructions or content doesn't make the task significantly harder. The "switch" itself is the hard part.

4. Different Tasks, Different Failures

The paper also showed that AI fails in different ways depending on the type of job:

Task Type A (Understanding Irony): The AI understood the meaning perfectly but failed to write the answer in the correct language.
Task Type B (Finding Specific Facts): The AI found the right fact but failed to follow strict formatting rules (like "only give the number, no sentences").
Task Type C (Updating Information): The AI got the language right but missed the actual update in the story.

5. Why This Matters

Previous tests often gave AI a single score, like "80% accuracy." This paper argues that a single score hides the truth. An AI might be a genius at understanding meaning but terrible at speaking the right language, or vice versa.

By separating these roles, the researchers found that the language the AI is asked to speak is the single biggest factor in whether it succeeds or fails.

Summary

The paper doesn't tell us how to fix the AI or how to use it in hospitals or schools. Instead, it acts like a mechanic's diagnostic tool. It tells us: "Don't just look at the overall score. If your AI is failing, check if it's struggling specifically with the language it has to output, because that's where the engine is stalling."

The study concludes that to truly understand multilingual AI, we need to stop treating language as a single blob and start looking at the specific role each language plays in the conversation.

Technical Summary: Disentangling Language Roles in Multilingual LLM Task Execution

1. Problem Statement

Multilingual Large Language Models (LLMs) are increasingly deployed in workflows where the instruction language, source content language, and required response language do not coincide. While existing benchmarks have expanded language coverage and instruction-following evaluation, they typically treat language as a monolithic variable or fail to isolate the specific roles of instruction, content, and response within a single evaluation design. Consequently, when a model fails, it remains unclear whether the error stems from task misunderstanding, content processing, target-language realization, constraint following, or interactions among these factors. This paper addresses the need for a controlled diagnostic framework to evaluate multilingual task execution as a structured interaction among three distinct language roles: $(L_{instr}, L_{content}, L_{resp})$ .

2. Methodology: MTM-Bench

The authors introduce MTM-Bench (Multilingual Triplet Mismatch Benchmark), a controlled benchmark designed to disentangle these roles.

Benchmark Construction

Design: The benchmark employs a fully crossed design using English (EN), Spanish (ES), and Chinese (ZH). It enumerates all $3^3 = 27$ possible language triplets.
Task Families: The benchmark consists of 2,430 instances per model across three distinct task families, each targeting a specific execution challenge:
1. Semantic Reversal (Family A): Tasks requiring the model to identify an underlying stance (e.g., irony, sarcasm) that contradicts surface wording.
2. Final-State Extraction (Family B): Tasks involving short administrative texts with updates, requiring the extraction of the final valid value (e.g., a rescheduled time or contact) while adhering to strict output formats.
3. Language Purity with Update Realization (Family C): Tasks requiring the model to describe a state update (e.g., a room change) in the target language, strictly localizing structural framing while allowing specific allowlisted proper nouns (e.g., "Zoom," "Teams").
Controlled Variables: Base items are rewritten rather than translated to ensure semantic stability across all 27 triplet variants. Metadata, names, and answer formats are held fixed where they define the task, while lexical material is localized according to the specific language slot.

Evaluation Protocol

The evaluation uses decomposed metrics to separate different failure modes:

Semantic Correctness (SC): Measures if the output captures the target meaning (e.g., correct polarity for reversal, correct final value for extraction).
Language Correctness (LC): A binary judgment on whether the response adheres to the required response language ( $L_{resp}$ ), treating numerals and allowlisted proper names as language-neutral.
Constraint Correctness (CC): Checks explicit output constraints (e.g., "output only," "single sentence").
Contamination Ratio (CTR): A continuous measure of non-target-language material in the output.
Joint Success (JS): The aggregate metric defined as $JS = SC \times LC \times CC$ , treating all three requirements as jointly necessary.

Scoring is validated through a targeted human audit covering boundary cases and routine outputs, with a component-level override rate of 18.2% for targeted cases, suggesting the automatic pipeline handles most routine cases consistently.

3. Key Contributions

Reframing Multilingual Execution: The paper reframes multilingual task execution as a triplet-based evaluation problem, varying instruction, content, and response languages independently rather than treating them as a single undifferentiated variable.
Fully Crossed Benchmark: It offers a controlled benchmark spanning 27 language triplets and three task families, scored with a decomposed protocol that isolates semantic correctness, language adherence, and constraint satisfaction.
Role-Specific Diagnostics: The fully crossed design enables marginal comparisons that aggregate benchmarks cannot isolate, revealing that degradation is organized by the role a language occupies, not merely by the count of mismatches.

4. Results

The authors evaluated 20 frontier and open-weight LLMs (including Claude, GPT, Gemini, DeepSeek, Qwen, and LLaMA families) on the full benchmark.

Overall Performance

Joint Success: Ranged from 0.537 to 0.840 across models (mean 0.745).
Component Metrics: Individual metrics were higher (mean LC: 0.905, SC: 0.893, CC: 0.880), indicating that satisfying all three requirements simultaneously is substantially harder than satisfying them in isolation.
Model Variance: The overall ranking did not reduce to any single component metric; models excelling in semantic correctness sometimes failed in constraint satisfaction, and vice versa.

Key Findings on Language Roles

Response Language Dominance: The response language is the dominant axis of variation. Joint success dropped significantly when the response language was Spanish ( $\Delta = -0.108$ ) or Chinese ( $\Delta = -0.113$ ) compared to English. In contrast, variations in instruction or content language had negligible effects ( $\Delta < 0.03$ ).
Mismatch Count vs. Role Position: The paper challenges the hypothesis that mismatch count is a monotonic predictor of difficulty. "Response-only" mismatch (one mismatch) and "Full mismatch" (three mismatches) yielded similar performance levels (0.694 vs. 0.705). This suggests that once the response language differs from the preceding context, additional mismatches in instruction or content add little explanatory power to the difficulty.
Task-Specific Bottlenecks:
- Semantic Reversal: High semantic correctness but limited by response-language realization, particularly for Chinese responses.
- Final-State Extraction: High language correctness but frequent failures in constraint satisfaction (e.g., violating "output-only" rules).
- Language Purity: High constraint satisfaction but lower semantic correctness due to difficulties in realizing updates without omitting or distorting information.

5. Significance and Claims

The paper claims that multilingual evaluation should track the role each language plays rather than collapsing distinct configurations into one aggregate score.

Diagnostic Value: The fully crossed design reveals that degradation is organized by the role a language occupies. The response-language role is the primary source of variation, while instruction and content roles play smaller, less consistent roles.
Non-Monotonic Difficulty: The finding that mismatch count is not a monotonic predictor of difficulty suggests that the cognitive or architectural burden of multilingual execution is not simply additive. The transition from a monolingual context to a mismatched response language constitutes the primary hurdle.
Limitations: The authors modestly note that these findings are controlled within the EN/ES/ZH setting and specific task families (short, administrative, and reversal scenarios). The results should be interpreted as a controlled diagnostic probe rather than a universal claim about all multilingual task execution or broader domain-level performance. The benchmark is designed for role-specific diagnosis rather than estimating broad domain-level performance.

In conclusion, MTM-Bench demonstrates that successful multilingual task execution requires more than general fluency; it requires the specific ability to realize answers in a target language that may differ from both the instruction and the source content, a capability that current models struggle with disproportionately compared to understanding or processing the other language roles.