💬 NLP

On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation

This paper identifies temperature-constrained Non-Deterministic Machine Translation (ND-MT) as a promising approach for addressing multimodality in translation, while revealing that current evaluation metrics suffer from a "Buckets Effect" that prioritizes worst-case candidates, leading to the proposal of a new strategy called ExpectoSample for robust system selection.

Original authors: Weichuan Wang, Mingyang Liu, Linqi Song, Chen Ma

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weichuan Wang, Mingyang Liu, Linqi Song, Chen Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Magic Dice" Translator

Imagine you have a translator app. Usually, when you type a sentence, it gives you one answer. It's like a robot that always picks the exact same path to get to the store. This is called Deterministic Translation (D-MT). It's safe, but sometimes it feels a bit robotic or misses the nuance that a sentence could be said in many different ways.

Recently, researchers discovered that modern AI translators have a "secret superpower." If you ask them to translate the same sentence multiple times, they don't always give the same answer. They might say:

"The cat sat on the mat."
"The feline rested on the rug."
"Upon the rug, the cat lay."

This is Non-Deterministic Translation (ND-MT). It's like the translator is rolling a pair of dice every time it speaks, giving you a variety of options. The paper explores two big questions:

Is this variety actually good? (Can it solve the problem of sentences having multiple meanings?)
How do we grade it? (If a student gives you 10 different answers, how do you decide if they passed or failed?)

1. The Good News: Solving the "Many Ways to Say It" Problem

In the real world, one sentence often has many valid translations. This is called Multimodality.

Example: "He saw the man with the telescope." Did he use the telescope to see the man? Or did the man have a telescope?

The Discovery:
The researchers found that when you turn on the "variety mode" (using a setting called Temperature), the AI generates a whole bucket of different translations.

Lexical Diversity: The words change (mat vs. rug vs. carpet).
Semantic Equivalence: The meaning stays the same.

The Catch (The Temperature Trap):
Think of Temperature like the heat on a stove.

Low Heat (Low Temp): The AI is careful. It gives you different words, but the meaning is perfect.
High Heat (High Temp): The AI gets too excited. It starts inventing words, changing the meaning, or even speaking gibberish.

The Verdict: Modern translators are "Temperature-Constrained." They can give you a great variety of high-quality options, but only if you keep the temperature low enough. If you crank it up too high, the quality collapses.

2. The Bad News: The "Bucket Effect" in Grading

Here is where things get tricky. How do you test a translator that gives you 10 different answers?

The Old Way (Deterministic):
You ask the AI for one answer, compare it to the "gold standard" answer, and give it a grade. Easy.

The New Problem (Non-Deterministic):
If you ask the AI for 10 answers, which one do you grade?

Do you grade the best one?
Do you grade the average one?
Do you grade the worst one?

The "Bucket Effect" Discovery:
The researchers found a surprising phenomenon they call the Buckets Effect.

Imagine a wooden bucket. The amount of water it can hold isn't determined by the longest plank; it's determined by the shortest plank.

Similarly, the researchers found that the overall ranking of a translation system is determined by its worst translation, not its best.

Even if an AI produces 9 perfect translations, if it produces one terrible, nonsensical translation, the system is considered "bad" by current grading tools.
Because the "worst" answer is random (you don't know which one it will be until you generate it), it's very hard to predict how good a system really is.

Why this matters:
If you use old grading methods, you might think System A is better than System B. But if you test them again with a different set of random "dice rolls," the rankings might flip because System B happened to generate a slightly worse "worst-case" scenario this time. The current grading tools are unreliable for these "variety" translators.

3. The Solution: "ExpectoSample" (The Reliable Filter)

Since we can't easily predict the "worst plank" in the bucket, the authors propose a new strategy called ExpectoSample.

Think of it like a Quality Control Inspector at a factory:

Step 1: Find the Right Ruler. Not all measuring tapes are good. Some are stretchy; some are rigid. The researchers tested many different grading tools (metrics) to see which ones give consistent results, no matter how many samples you take. They found a few "reliable rulers."
Step 2: The Quick Test. Instead of generating 1,000 translations to find the worst one (which is too expensive and slow), they suggest using a small, random sample (like 10 translations) and using those "reliable rulers" to pick the best system.

This strategy helps companies choose the most robust AI translator without needing to test every single possible outcome.

Summary in a Nutshell

The Potential: Modern AI translators can give you a variety of high-quality options (like a chef offering a few different plating styles for the same dish), which is great for handling complex sentences.
The Constraint: This only works if you keep the "temperature" setting low. If you get too creative, the meaning breaks.
The Problem: We don't know how to grade these systems properly yet. Current tests are fooled by the "worst" random answer the AI gives, making it look like the system is worse than it actually is (The Bucket Effect).
The Fix: The authors created a new method (ExpectoSample) to find the most reliable ways to test these systems, ensuring we pick the ones that are consistently good, even when they are being a bit unpredictable.

The Bottom Line: Non-deterministic translation is a powerful tool, but we need new rules of the road to measure it safely and fairly.

1. Problem Statement

The paper addresses two critical gaps in the current landscape of Machine Translation (MT):

The Multimodality Challenge: A single source sentence often has multiple valid translations (lexical diversity) that share the same meaning (semantic equivalence). Traditional Deterministic MT (D-MT) systems, which generate a single output via greedy decoding, struggle to capture this diversity, often leading to a "one-size-fits-all" translation that may not suit all contexts.
Evaluation Limitations: Existing evaluation frameworks are designed for D-MT. When applied to Non-Deterministic MT (ND-MT), where models generate multiple candidates via sampling, these frameworks fail. Specifically, standard metrics (like BLEU or COMET) yield inconsistent rankings when applied to groups of candidates, and they cannot effectively measure the trade-off between lexical diversity and semantic fidelity without gold-standard references (which are scarce for ND-MT).

2. Methodology

The authors conducted a systematic investigation involving 22 modern MT systems (including encoder-decoder models like mBART, NLLB, and decoder-only LLMs like Llama, Qwen, and DeepSeek) across six language directions (e.g., En↔Zh, En↔De, En↔Ru).

Key Experimental Components:

Temperature-Constrained Sampling: Instead of greedy decoding, the authors used sampling-based decoding with adjustable temperatures (focusing on $T=0.5$ ) to generate multiple candidates ( $K=10, 20, 50$ ) per source sentence.
Metric Selection:
- Lexical Metrics: BLEU, METEOR, ROUGE, TER, ChrF++.
- Semantic Metrics: COMET, BERTScore, BLEURT, LASER, XNLI.
- Proposed Metric (GLVS): A new Group Lexical Variance Score designed to measure lexical diversity without relying on reference translations. It calculates the frequency distribution of unique tokens across a candidate set; a lower score indicates higher diversity.
Group-Level Aggregation: To evaluate ND-MT, the authors aggregated scores across candidate groups using statistics: min, max, mean, random, and standard deviation (std).
Temperature Analysis: They varied temperature settings to observe the trade-off between generating diverse words and maintaining semantic equivalence.

3. Key Contributions

Characterization of Modern MT as "Temperature-Constrained ND-MT": The paper establishes that modern MT systems can effectively address multimodality by generating lexically diverse candidates while maintaining semantic equivalence, provided the temperature is carefully constrained (typically low to moderate).
Discovery of the "Buckets Effect": The authors identified a critical phenomenon in ND-MT evaluation where the system ranking is dominated by the worst-quality candidate (the "shortest plank" in the bucket). Automatic metrics correlate strongly with the minimum score of a candidate set rather than the mean or maximum, making system evaluation highly sensitive to stochastic generation failures.
Proposal of ExpectoSample: A two-stage strategy to mitigate evaluation instability:
- Stage 1 (Metric Filtering): Identify metrics that maintain consistent system rankings across different sampling sizes (filtering out unstable metrics).
- Stage 2 (System Selection): Use these reliable metrics with small sample sizes to select robust ND-MT systems for deployment.
Comprehensive Benchmarking: A large-scale evaluation of 22 systems using 11,947 source cases, providing a new dataset and codebase for future research.

4. Key Results

Lexical Diversity vs. Semantic Equivalence:
- ND-MT systems generate significantly higher lexical diversity than D-MT (detected by GLVS, whereas traditional metrics like BLEU often fail to detect this).
- Semantic Equivalence is Temperature-Sensitive: While higher temperatures increase lexical diversity, they degrade semantic equivalence. Low-to-moderate temperatures (e.g., $T=0.5$ ) offer the best balance, preserving meaning while providing variety.
The Buckets Effect:
- Evaluation rankings for ND-MT systems are almost perfectly correlated with the minimum (worst-case) score of the generated candidates.
- The maximum (best-case) scores show weak correlation with overall system ranking, indicating that a model's "lucky" best outputs do not predict its general reliability.
- Standard deviation (stability) is a strong indicator of system quality; systems with lower variance are ranked higher.
Potential for Higher Quality:
- ND-MT systems possess a "high-quality upper bound" that exceeds the performance of their deterministic counterparts. This suggests that with better selection strategies (e.g., Minimum Bayes Risk decoding), ND-MT could produce translations superior to current state-of-the-art D-MT.
Metric Reliability:
- Metrics like ChrF++, COMET-DA, and COMET-Kiwi were found to be robust across different sampling sizes, whereas others (like BLEU in certain contexts) showed instability.

5. Significance and Implications

Paradigm Shift in Evaluation: The paper argues that evaluating MT systems solely on single-output (greedy) generation is insufficient for the era of LLMs. It necessitates a shift toward evaluating the distribution of outputs, acknowledging that the "worst-case" scenario defines the reliability of a non-deterministic system.
Practical Deployment: The ExpectoSample strategy provides a practical framework for developers to select stable translation models without incurring the prohibitive costs of massive sampling or human evaluation.
Addressing Multimodality: By validating that ND-MT can satisfy both lexical diversity and semantic equivalence, the paper offers a solution to the long-standing multimodality problem, enabling applications that require varied yet accurate translations (e.g., creative writing, localization, or user-customized chatbots).
Temperature as a Control Knob: The findings emphasize that temperature is not just a hyperparameter for randomness but a critical constraint that must be tuned to balance diversity and fidelity in real-world applications.

In conclusion, this work bridges the gap between the non-deterministic capabilities of modern LLMs and the rigorous evaluation standards required for Machine Translation, proposing both a theoretical understanding of the "Buckets Effect" and a practical methodology for robust system selection.