💬 NLP

Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LM

This paper presents an empirical evaluation of cross-family speculative decoding for Polish language models on Apple Silicon, demonstrating that while context-aware token translation improves acceptance rates, unified memory bandwidth constraints and mismatched tokenizers limit throughput gains, particularly when specialized draft models underperform general-purpose alternatives.

Original authors: Krzysztof Fonal

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Krzysztof Fonal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Drafting Assistant" vs. The "Strict Editor"

Imagine you are writing a long, complex story in Polish. You have a Strict Editor (the big AI model, Bielik 11B) who is incredibly smart, knows the rules perfectly, and writes perfect Polish. But, the Editor is slow. Every time they write a word, they have to walk to the library, find the book, read the page, and come back. This takes a lot of time.

Speculative Decoding is a technique where you hire a Drafting Assistant (a smaller, faster AI model) to guess the next few words for you.

The Assistant quickly scribbles down 2 or 4 words.
The Editor looks at those words all at once.
If the Editor agrees, "Great! Keep them!" and moves on.
If the Editor disagrees, "No, that's wrong," they cross it out and write the correct word themselves.

The Goal: If the Assistant is good enough, the Editor spends less time walking to the library because they verify multiple words in a single trip. This should make the whole process much faster.

The Problem: Speaking Different "Dialects"

Usually, this works great if the Assistant and the Editor speak the exact same language and use the same dictionary. But in this paper, the researchers tried to pair a Polish-specific Editor with three different Assistants:

A Polish-specific Assistant (Bielik 1.5B).
A general Chinese/English Assistant (Qwen).
A general English Assistant (Llama).

The Catch: These Assistants use different dictionaries (tokenizers).

The Editor might see the word "warszawa" as one single token.
The Assistant might see it as "w" + "arszawa".

If the Assistant guesses "w" and the Editor expects "warszawa", the Editor gets confused. It's like the Assistant speaking in a dialect where "apple" is called "pomme," but the Editor only understands "apple." If they don't translate the words correctly, the Editor rejects the guess, and the speedup is lost.

The Solution: The "Context Translator"

The researchers built a special Translator (called Universal Assisted Generation or UAG) to fix this.

Naive Translation: Just translating the word directly. This failed often because the Assistant didn't know the context. (e.g., Is "w" the word "in" or the start of "wine"?).
Context-Aware Translation: The Translator looks at the last few words the Editor already accepted to understand the context before translating the Assistant's guess. This is like saying, "Oh, you're talking about a city, so 'w' means 'in Warsaw'."

The Result: This "Context Translator" was a game-changer. It allowed the different AI models to work together, significantly improving how many guesses the Editor accepted.

The Surprise: The "Polish Specialist" Lost

You might think the Polish-specific Assistant (Bielik 1.5B) would be the best at guessing Polish text. Surprisingly, it wasn't.

The general-purpose Assistants (Qwen and Llama) actually guessed better than the Polish specialist.
Why? The Polish specialist used a very complex, custom dictionary designed specifically for Polish grammar. This made it hard to translate its guesses into the Editor's dictionary without errors. The general Assistants used simpler, more standard dictionaries that were easier to translate, leading to fewer mistakes.

The Hardware Reality: The "Narrow Hallway"

This is the most important part of the paper. The researchers tested this on an Apple Mac (M2 Pro).

On Super-Computers (NVIDIA GPUs): The "hallway" (memory bus) is wide and fast. The Editor can check 4 words almost as fast as checking 1. So, asking the Assistant to guess 4 words is a huge win.
On Apple Macs: The "hallway" is narrow and slow. The Editor has to wait for the data to arrive before they can do anything.
- Even though the Assistant is fast, the Editor is still stuck waiting for the data to load from memory.
- The Finding: Asking the Assistant to guess 4 words actually made things slower than just writing one word at a time. The time spent waiting for the data to load outweighed the benefit of checking multiple words.
- The Sweet Spot: The only time it worked was when the Assistant guessed 2 words. Anything more was too much overhead for the narrow hallway.

The Energy Bonus: The "Eco-Friendly" Laptop

The paper also looked at energy.

A massive server farm (NVIDIA) uses a lot of electricity to generate words.
The Apple Mac uses a tiny fraction of that power.
Analogy: It's like driving a massive semi-truck (Server) vs. a hybrid sedan (Mac). The truck is faster, but the sedan uses way less gas per mile. For a single person working at home, the Mac is much more efficient and keeps your data private (no need to send your documents to a cloud server).

Summary of Key Takeaways

Translation is Key: To make different AI models work together, you need a smart translator that looks at the context, not just a direct word swap.
Don't Over-Guess: On consumer laptops (like Macs), asking the AI to guess too many words at once (more than 2) actually slows you down because of memory limits.
General is Better than Specialized (Sometimes): For this specific setup, a general AI model was a better "assistant" than a specialized Polish model because its language was easier to translate.
Privacy & Efficiency: Running these models locally on a Mac is a great way to keep your data private and save energy, even if it's not as fast as a supercomputer.

In short: The researchers figured out how to make different AI models talk to each other on a Mac. They found that while it works, you have to be careful not to ask the "assistant" to do too much at once, or the "editor" gets overwhelmed by the slow data delivery!

1. Problem Statement

The paper addresses three interconnected challenges in the deployment of Large Language Models (LLMs) on consumer-grade hardware:

Cross-Tokenizer Incompatibility: Standard speculative decoding requires draft and target models to share the same tokenizer. However, the Polish language ecosystem relies on the Bielik family, which is not tokenizer-homogeneous. The target model, Bielik 11B, uses a Mistral-derived tokenizer, while the smaller draft model, Bielik 1.5B, uses a custom Polish tokenizer (APT4) based on Qwen2.5. This mismatch prevents standard speculative decoding.
Hardware Constraints: Apple Silicon (e.g., M2 Pro) uses a Unified Memory Architecture (UMA) with significantly lower memory bandwidth (~~200 GB/s) compared to discrete NVIDIA GPUs (~~2 TB/s). Theoretical speedups from speculative decoding assume that verifying $k$ tokens costs the same as generating one (due to memory bandwidth bottlenecks). It is unclear if this assumption holds on UMA, where both draft and target models compete for the same bus, potentially making sequential draft passes prohibitively expensive.
Lack of Empirical Data: There is no systematic evaluation of speculative decoding for morphologically rich, inflected languages like Polish, nor for cross-family model pairs on Apple Silicon.

2. Methodology

The authors extended the MLX-LM inference framework (Apple's native LLM library) to support Universal Assisted Generation (UAG), enabling cross-tokenizer speculative decoding.

Models:
- Target: Bielik 11B-Instruct (Mistral architecture, 8-bit quantized).
- Drafts: Three cross-family models (4-bit quantized):
  1. Bielik 1.5B (Qwen2.5 base, APT4 tokenizer).
  2. Qwen2.5-1.5B (Qwen tokenizer).
  3. Llama 3.2-1B (Llama tokenizer).
Translation Strategies:
- Naive Translation: Decodes draft tokens to text and re-encodes them into the target vocabulary without context.
- Context-Aware Translation: A novel implementation for MLX-LM that prepends a window of $p$ previously accepted tokens (prefix) before re-encoding. This resolves boundary ambiguities caused by different tokenization strategies (e.g., how Polish diacritics or function words like "w" are segmented).
Experimental Setup:
- Hardware: Apple M2 Pro (32 GB Unified Memory, 200 GB/s bandwidth).
- Datasets: Polish Wikipedia (structured/factual), pl_alpaca (instruction-following), and synthetic short questions.
- Variables: Draft lengths $k \in \{2, 4, 6\}$ and translation strategies.

3. Key Contributions

Implementation of UAG in MLX-LM: The first port of Universal Assisted Generation to Apple's MLX framework, supporting both naive and context-aware token translation strategies.
Empirical Characterization of Cross-Family Decoding: The first systematic study showing that for Polish, general-purpose drafters (Qwen, Llama) outperform the Polish-specialized drafter (Bielik 1.5B) due to better tokenizer alignment with the target's vocabulary structure, despite the latter being "same-family."
Hardware-Aware Cost Model: The authors derived a parametric speedup formula demonstrating that on Apple Silicon, the cost of $k$ draft passes does not amortize as theory predicts. They identified that the draft overhead is linear in $k$ while the benefit is sub-linear, leading to a "break-even" acceptance rate that rises sharply with $k$ .
Content-Type Analysis: Demonstrated that speculative decoding is highly content-dependent, yielding speedups on structured/repetitive text (Wikipedia) but failing on variable instruction-following tasks.

4. Key Results

A. Token Acceptance Rates

Context-Aware is Essential: Context-aware translation consistently achieved the highest acceptance rates (30–45%). Naive translation performed poorly (often <15%), and "no translation" was only viable for structured text where token IDs coincidentally matched.
The "Polish Paradox": The Polish-specialized Bielik 1.5B drafter achieved the lowest acceptance rates (approx. 22–31%) compared to Qwen (41–44%) and Llama (36–42%). This is attributed to the APT4 tokenizer's aggressive morphological segmentation, which creates a mismatch with the Mistral-based target tokenizer that general-purpose models handle better via context-aware translation.

B. Throughput and Speedup

Optimal Draft Length ( $k=2$ ): Speedups were only achievable at $k=2$ $k = 2$ .
- On Wikipedia (structured text), Qwen and Llama achieved 1.06× speedup.
- On instruction-following data, speedups were marginal or negative.
Failure at Higher $k$ : Increasing $k$ $k$ to 4 or 6 resulted in severe performance degradation (0.40×–0.60× baseline).
- Break-Even Threshold: To achieve a speedup at $k=4$ , the acceptance rate needed to exceed 77–90%, a threshold never reached in practice.
- Cause: On the M2 Pro, the memory bandwidth is the bottleneck. Verifying $k$ tokens is fast, but generating $k$ draft tokens requires $k$ sequential forward passes through the draft model, streaming weights over the shared bus. The cost of these $k$ passes outweighs the benefit of batched verification.

C. Energy Efficiency

Despite lower raw throughput compared to high-end NVIDIA GPUs, the M2 Pro is highly energy-efficient for single-user inference.
M2 Pro: ~2.0–2.7 Joules/token.
Comparison: Comparable to the datacenter-grade NVIDIA A100 (~~2.3 J/token) and significantly more efficient than consumer RTX 4090 (~~5.6–7.5 J/token) for batch size 1.

5. Significance and Implications

Viability for Low-Resource Languages: The study proves that cross-family speculative decoding is viable for Polish (and likely other inflected languages) on consumer hardware, provided context-aware translation is used. Users do not need a Polish-specific small model; general-purpose models like Qwen or Llama serve as effective drafters.
Hardware Reality Check: The paper challenges the standard assumption that speculative decoding is universally beneficial. On unified memory architectures with limited bandwidth, longer draft sequences ( $k > 2$ ) are counterproductive. The "sweet spot" is short drafts ( $k=2$ ) on structured content.
Deployment Guidelines: For practitioners deploying Polish LLMs on Apple Silicon:
1. Use Context-Aware Translation (prefix window) to handle tokenizer mismatches.
2. Set $k=2$ ; do not increase draft length.
3. Expect speedups primarily on structured/repetitive text; open-ended generation may see no benefit or a slowdown.
4. Prioritize energy efficiency and data privacy (local inference) over raw speed, as the M2 Pro offers competitive J/token metrics compared to datacenter GPUs.

Conclusion

This work bridges a critical gap in LLM deployment for under-resourced languages on consumer hardware. By extending MLX-LM with UAG, the authors enabled cross-family speculative decoding for Polish, revealing that while the technique is viable, its success is strictly bounded by memory bandwidth constraints and content predictability. The findings provide a concrete, hardware-aware framework for optimizing local LLM inference on Apple Silicon.