Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series

This paper introduces the Bielik v3 7B and 11B models, which achieve significant performance improvements for the Polish language by replacing universal tokenizers with a dedicated, optimized vocabulary and employing a comprehensive training pipeline that includes FOCUS-based initialization, multi-stage pretraining, and advanced alignment techniques.

The Gist

Imagine you have a brilliant, multilingual librarian named Bielik. This librarian is incredibly smart, knows a lot about the world, and can chat in many languages. However, there's a problem: Bielik was originally trained using a "universal dictionary" designed for the whole world.

When Bielik tries to speak Polish, this universal dictionary is inefficient. It's like trying to write a complex Polish sentence using a dictionary where every single letter is a separate word. To say "I am going to the store," Bielik might have to use 10 tiny, fragmented pieces of information instead of 4 clear words. This makes Bielik slow, expensive to run, and limits how much he can remember at once (his "context window").

This paper is the story of how the creators gave Bielik a custom-made Polish dictionary to make him faster, smarter, and more efficient, without making him forget everything else he knows.

Here is the breakdown of their journey, using simple analogies:

1. The Problem: The "Universal Dictionary" Glitch

Think of the original tokenizer (the dictionary Bielik uses) as a Swiss Army Knife. It's great because it has a screwdriver, a blade, and a corkscrew for everyone. But if you are a Polish chef trying to chop onions, you don't need a corkscrew; you need a sharp, specialized knife.

• The Issue: The Swiss Army Knife (universal tokenizer) breaks Polish words into too many tiny, useless pieces. This is called a high "fertility ratio." It's like trying to fill a swimming pool with a teaspoon instead of a hose. You waste time and energy just moving the water.

2. The Solution: The "Custom Polish Knife" (APT4 Tokenizer)

The team created a new dictionary specifically for Polish, called APT4.

• The Magic: Instead of breaking a Polish word into 3 or 4 tiny pieces, this new dictionary treats the whole word as one smooth unit.
• The Result: Bielik can now "think" in Polish twice as fast. He uses half the memory to say the same thing, and he can remember twice as much of the conversation at once. It's like upgrading from a bicycle to a sports car for Polish tasks, while still being able to drive on English roads.

3. The Danger: "Amnesia" (Catastrophic Forgetting)

Here's the tricky part. If you suddenly swap a librarian's entire dictionary while they are working, they might get confused and forget how to read any language, including the ones they already knew. This is called Catastrophic Forgetting.

• The Fix (FOCUS): To prevent this, the team used a clever technique called FOCUS. Imagine you are teaching a student a new alphabet. Instead of making them start from scratch, you show them how the new letters overlap with the old ones. You say, "This new letter 'A' is just a combination of the old 'A' and 'B'."
• The Process: They used math to map the old words to the new words carefully, ensuring Bielik didn't lose his memory of English or his general reasoning skills.

4. The Training: "Re-education Camp" (Two-Stage Pipeline)

You can't just swap the dictionary and expect Bielik to be perfect immediately. They had to retrain him in two phases:

• Phase 1 (The Warm-up): They froze most of Bielik's brain and only let him practice with the new dictionary on a small amount of text. It was like teaching him how to hold the new knife without letting him cook a full meal yet.
• Phase 2 (The Full Meal): Once he got comfortable, they unfroze his whole brain and let him practice on a massive amount of Polish text (16 billion words!). This allowed him to fully adapt his "muscle memory" to the new, efficient dictionary.

5. The Polish: "Fine-Tuning" (SFT, DPO, GRPO)

After learning the new dictionary, Bielik needed to learn how to be a good conversationalist again.

• SFT (Supervised Fine-Tuning): They taught him how to follow instructions and chat politely.
• DPO (Preference Optimization): They showed him examples of "good" answers vs. "bad" answers so he learned to prefer being helpful and honest.
• GRPO (Reinforcement Learning): They gave him math and logic puzzles to solve, rewarding him when he got the steps right, sharpening his reasoning skills.

6. The Results: Did it Work?

The team put Bielik through a series of tests (like a report card):

• Polish Tests: He aced them! In fact, on some tests involving complex Polish logic and emotions, the new "Polish-optimized" Bielik actually scored higher than the original version.
• English Tests: He didn't lose his English skills. He still speaks English very well, proving the "amnesia" fix worked.
• Speed & Cost: Because he uses fewer "tokens" (pieces of words) to say the same thing, he is faster and cheaper to run.

The Bottom Line

The creators of Bielik v3 took a powerful, general-purpose AI and gave it a specialized Polish brain. They did this without making him forget his other languages or his smarts.

Think of it this way: Before, Bielik was a brilliant polyglot who spoke Polish with a stutter because he was using the wrong dictionary. Now, he speaks Polish fluently, quickly, and with perfect grammar, while still being able to chat in English just as well as before. And the best part? They gave away the blueprints (the model weights) for free so everyone can use this super-efficient Polish AI.

Technical Summary

1. Problem Statement

The paper addresses a fundamental inefficiency in general-purpose Large Language Models (LLMs) when applied to morphologically rich languages like Polish.

• Universal Tokenizer Limitations: Standard tokenizers (e.g., those used in Mistral or Llama) are designed for multilingual coverage. For Polish, this results in a high fertility ratio (the average number of tokens required to represent a word).
• Consequences: High fertility leads to:
  • Reduced information density within the context window.
  • Increased inference costs and latency.
  • Restricted effective context capacity for Polish text.
• The Trade-off: While increasing vocabulary size can reduce fragmentation, it drastically increases model size and memory consumption, making it inefficient for monolingual or language-specific applications.

2. Methodology

The authors introduce the Bielik v3 PL series (7B and 11B parameters), which retains the Mistral-based architecture but replaces the universal tokenizer with a dedicated Polish-optimized APT4 tokenizer. The core challenge was adapting a pretrained model to a new vocabulary without losing previously acquired knowledge (catastrophic forgetting).

A. Tokenizer Design

• Vocabulary: The new tokenizer uses a vocabulary size of 32,000 tokens, comparable to the original 32,128-token vocabulary, ensuring fair comparison.
• Optimization: The APT4 tokenizer (an evolution of APT3) is specifically tuned for Polish morphology, reducing the tokens-per-word (TpW) metric significantly compared to universal tokenizers.

B. Vocabulary Adaptation Strategy (FOCUS)

To transfer knowledge from the original vocabulary to the new Polish-optimized one, the authors employed the FOCUS (Fast Overlapping Token Combinations Using Sparsemax) framework.

• Mechanism: FOCUS represents new tokens as sparse linear combinations of original tokens based on semantic similarity in an auxiliary embedding space.
• Comparison: The authors evaluated FOCUS against other methods (Random Initialization, Frequency-based Transfer, Linear Transformation, WECHSEL, MATT, OFA, RAMEN). FOCUS demonstrated the lowest training loss and best stability.

C. Training Pipeline

The adaptation process involved a multi-stage pipeline:

1. Continued Pretraining (Two Stages):
   • Stage 1 (Partial Freezing): Trained on 4B tokens. Only the input embedding layer, the language modeling head (lm_head), and four boundary transformer layers (two lowest, two highest) were updated. This stabilized the mapping between the new token space and the frozen internal representations.
   • Stage 2 (Full Adaptation): All parameters were unfrozen, and the model was trained on an additional 16B tokens to globally adjust weights to Polish linguistic characteristics.
2. Post-Training Alignment:
   • Supervised Fine-Tuning (SFT): 3 epochs on ~20M instruction-response pairs (Polish/English).
   • Direct Preference Optimization (DPO-P): 3 epochs on 114k preference-labeled examples to reduce hallucinations.
   • Reinforcement Learning (GRPO): Group Relative Policy Optimization using verifiable rewards (Math, Logic, STEM) on 143k examples to enhance reasoning.

3. Key Contributions

• Tokenizer Replacement: A successful method for swapping the tokenizer of a pretrained LLM using FOCUS, mitigating catastrophic forgetting while improving language-specific efficiency.
• Efficiency Gains: The new tokenizer reduces the fertility ratio for Polish text from 3.22 tokens/word (original Bielik v3) to 1.62 tokens/word (Bielik v3 PL), effectively doubling the context capacity for Polish without changing the model architecture or parameter count.
• Open Release: Both the 7B and 11B models are released under the Apache 2.0 license.
• Scalability: The methodology was applied to both the flagship 11B model (scaled via Depth Up-Scaling) and the compressed 7B Minitron variant (derived via pruning and distillation).

4. Results and Evaluation

The models were evaluated across nine benchmarks covering Polish-specific tasks, multilingual capabilities, and reasoning.

• Polish Performance (Preservation & Improvement):

  • Open PL LLM Leaderboard: The original 11B model scored 65.93. The Polish-optimized 11B PL variant scored 64.11, demonstrating that performance was largely preserved despite the tokenizer change.
  • Complex Polish Text Understanding (CPTUB): The PL variant actually surpassed the original, scoring 3.80 vs. 3.73, particularly in understanding implicatures and phraseology.
  • Polish EQ-Bench: The PL 11B scored 71.15, nearly matching the original's 71.20.
  • Polish Medical Leaderboard: The PL 11B achieved 48.42%, showing strong domain-specific reasoning.

• Multilingual & English Performance:

  • Open LLM Leaderboard (English): The PL 11B scored 71.49, very close to the original's 72.45, indicating that English capabilities were not significantly degraded.
  • Belebele (Reading Comprehension): The PL 11B scored 77.41 on the European average (vs. 82.98 for the original), showing a slight trade-off where Polish optimization slightly reduced broad multilingual scores, but Polish-specific scores remained high (81.22).
  • FLORES (Translation): The PL 11B achieved a BLEU score of 17.82 (vs. 19.22 for original), with strong bidirectional translation capabilities.

• Efficiency Metrics:

  • Tokens per Word (Polish): Reduced from 3.22 to 1.62.
  • Characters per Token: Improved from 2.40 to 4.78, indicating denser encoding.

5. Significance

This work provides a reproducible blueprint for adapting multilingual LLMs to specific languages where morphological complexity hinders the efficiency of universal tokenizers.

• Cost Reduction: By nearly doubling the effective context window for Polish without increasing model parameters, inference costs are significantly lowered.
• Performance Parity: The study proves that specialized tokenization does not require retraining from scratch; it can be achieved through structured vocabulary transfer (FOCUS) and staged pretraining while maintaining performance comparable to, or better than, general-purpose models.
• European AI Sovereignty: The release of high-quality, open-weight models specifically optimized for Polish contributes to the broader European effort to develop accessible, linguistically diverse AI systems.