💬 NLP

Speculative Decoding Across Languages

This paper investigates methods to enhance speculative decoding efficiency for non-English languages by comparing task-specific finetuning, monolingual finetuning, and n-gram models, finding that while task-specific distillation improves efficiency but lacks generalization, n-gram models consistently deliver significant speed-ups due to their rapid draft generation despite lower acceptance rates.

Original authors: Nirajan Paudel, Michael Ginn, Luc De Nardi, Alexis Palmer

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nirajan Paudel, Michael Ginn, Luc De Nardi, Alexis Palmer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a long story or translate a document in a language that isn't your native tongue. You have a very smart, but slow, "Master Writer" (the Large Language Model) who can do the job perfectly, but they take a long time to think of every single word before writing it down.

To speed things up, you hire a "Speedy Assistant" (the Draft Model). The Assistant guesses the next few words quickly, and the Master Writer just checks them. If the guesses are right, the Master Writer accepts them all at once, saving a lot of time. This is called Speculative Decoding.

However, the paper finds a major problem: The Assistant is great at English, but terrible at other languages.

Here is a breakdown of what the researchers tried to fix this, using simple analogies:

The Problem: The "One-Size-Fits-None" Assistant

When the researchers tried to use this speed-up trick for 11 different languages (like Nepali, Cherokee, or Yoruba), the standard "Speedy Assistant" (a small AI model) kept guessing wrong. Because the Assistant guessed wrong so often, the Master Writer had to reject the guesses and start over. This made the process slower than just letting the Master Writer write alone.

It's like hiring a tour guide who knows London perfectly but has never visited Nepal. If you ask them to guide you through Kathmandu, they will get lost constantly, and you'll waste more time correcting them than if you had just walked yourself.

The Three Solutions They Tested

The researchers tried three different ways to train a better Assistant for these specific languages:

1. The "Specialized Intern" (Task-Specific Distillation)

They took the Master Writer and taught the Assistant specifically how to translate English into the target language.

The Result: This worked great for translation. The Assistant became very good at guessing the next word for translation tasks.
The Catch: This Assistant was like a specialist who only knows how to translate. When the researchers asked the Assistant to write a story (a completely different task) in that same language, the Assistant failed miserably. It couldn't generalize. It was too specialized.

2. The "Generalist Reader" (General Domain Distillation)

Instead of teaching the Assistant only translation, they fed it huge amounts of general text in that language (news, books, etc.) and asked it to learn the patterns.

The Result: This didn't help much. The Assistant didn't get significantly better at guessing words compared to the untrained version. It was like reading a library of books but not learning how to write a story or translate a sentence effectively.

3. The "Super-Fast Calculator" (N-Gram Models)

Instead of using a complex AI model, they used a very simple statistical method called an N-gram model. Think of this not as a "smart" AI, but as a super-fast calculator that looks at the last few words and says, "Statistically, the next word is usually 'the' or 'is'."

The Result: This was the surprise winner.
- Accuracy: It wasn't very smart. It guessed wrong more often than the AI models.
- Speed: It was incredibly fast. Because it was so simple, it could generate guesses almost instantly.
- The Outcome: Even though it guessed wrong more often, the sheer speed of its guesses meant the whole system finished the job much faster. It was like having a child who guesses words randomly but types 100 words a second, versus a professor who types one word a second but is always right. The child actually got the job done faster overall.

The Big Takeaway

The paper concludes that for non-English languages, the "smart but slow" AI assistants often fail to speed things up.

If you need to do one specific job (like translation), you can train a specialized AI assistant, but it won't help you with other jobs (like writing stories).
If you want consistent speed across different tasks and languages, the best option is actually the simple, fast, statistical model (the N-gram). It's not the smartest, but it's so fast that it beats the smarter, slower models every time.

In short: For languages other than English, sometimes the "dumb but fast" approach is the best way to get things done.

Technical Summary: Speculative Decoding Across Languages

Problem Statement

Speculative decoding is a technique designed to accelerate Large Language Model (LLM) inference by using a lightweight "draft" model to propose multiple tokens, which are then verified in parallel by a larger "verifier" model. While effective in English, this paper identifies a critical limitation: small draft models often possess disproportionately poor multilingual capabilities compared to their larger counterparts. Consequently, when generating text in non-English languages, the acceptance rate of drafted tokens drops significantly. This low acceptance rate negates the speed-up benefits of speculative decoding, sometimes resulting in generation speeds slower than standard autoregressive decoding. This disparity exacerbates existing inequalities in language technology, particularly for low-resource languages where tokenization biases may further increase token counts.

Methodology

The authors evaluate speculative decoding efficiency across eleven languages (including Amharic, Berber, Cherokee, Nepali, and Yoruba) using the Qwen 3.5 model family. The verifier is a 9B parameter model, and the baseline draft model is a 0.8B parameter model of the same family. The study compares three distinct strategies for training or constructing draft models:

Task-Specific Distillation: Fine-tuning the draft model on parallel translation data (English to target language) to minimize KL divergence with the verifier's distribution on specific translation prompts.
General Domain Distillation: Fine-tuning the draft model on unlabeled, monolingual corpora of the target language without task-specific prompts.
N-Gram Modeling: Training simple n-gram models (specifically bigrams) on the same monolingual corpora, using the Qwen tokenizer.

The evaluation covers two tasks:

In-Domain: Machine Translation (English to target language).
Out-of-Domain: Open-ended story generation in the target language (to test generalization without specific training on story data).

Performance is measured via Acceptance Rate ( $\alpha$ ) and Speed-up Factor ( $f$ ), the latter calculated based on the ratio of forward pass times between the draft and verifier models.

Key Findings

1. Baseline Ineffectiveness

Standard speculative decoding using an unmodified small draft model yields poor results for non-English languages. Average acceptance rates across the eleven languages were approximately 0.40, resulting in a speed-up factor near 1.02 $\times$ , rendering the technique largely ineffective.

2. Distillation: Effective In-Domain, Poor Generalization

Task-Specific Distillation: Fine-tuning on translation data significantly improved acceptance rates for the translation task ( $\bar{\alpha} = 0.60$ ), achieving a speed-up of 1.28 $\times$ . However, these models failed to generalize to the story generation task, where performance deteriorated below the baseline ( $\bar{\alpha} = 0.43$ , $\bar{f} = 1.03\times$ ).
General Domain Distillation: Distilling on monolingual corpora provided marginal improvements over the baseline for translation but did not consistently outperform the unmodified baseline across all languages.

3. N-Gram Models: Superior Speed via Low Cost

Despite having the lowest acceptance rates among the tested methods ( $\bar{\alpha} = 0.24$ for story generation, $0.30$ for translation), n-gram models consistently provided the highest speed-up factors (up to 1.39 $\times$ for story generation and 1.30 $\times$ average).

Reasoning: The n-gram models are computationally negligible (forward pass time $\approx 0.001$ s) compared to neural draft models ( $\approx 0.033$ s for the 0.8B model). This favorable cost ratio ( $c = t_{draft}/t_{target}$ ) allows the system to achieve significant speed-ups even with lower acceptance rates.

4. Lack of Correlation with Translation Quality

The study found no significant correlation between a model's translation capability (measured by chrF score) and the acceptance rate of the draft model, challenging the hypothesis that acceptance rates are simply a proxy for the model's proficiency in a specific language.

Significance and Claims

The paper claims that standard speculative decoding is often detrimental or useless for non-English languages when using standard small neural draft models. The primary contribution is the demonstration that simple n-gram models, despite their low acceptance rates, are a highly viable and often superior alternative for multilingual speculative decoding due to their minimal inference cost.

The authors argue that for non-English generation, the optimal strategy may not be to train complex neural draft models, but rather to dynamically switch to n-gram draft models when a monolingual corpus is available. This approach offers a practical path to mitigating generation speed disparities for disadvantaged languages without requiring extensive task-specific training data or complex distillation pipelines.

The work acknowledges limitations, noting that results are specific to the Qwen 3.5 family and the eleven languages tested, and that the generated story outputs were of low quality, though the focus remained strictly on inference speed metrics.