💻 computer science

RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

This paper introduces RightNow-Arabic-0.5B-Turbo, an open-source, 518M-parameter Arabic-specialized language model built on Qwen2.5-0.5B through vocabulary injection and extensive fine-tuning, which achieves state-of-the-art performance among sub-1B models while enabling efficient edge deployment with quantization to under 400 MB.

Original authors: Jaber Jaber, Osama Jaber

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaber Jaber, Osama Jaber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot assistant named Qwen. It's tiny and fast, fitting easily on a laptop or even a phone, but it treats Arabic like just one of 100 languages it knows a little bit about. It's not fluent; it stutters and needs to break Arabic words into many tiny, awkward pieces to understand them.

On the other end of the spectrum, there are giant Arabic robots (like Jais or SILMA). These are incredibly fluent and knowledgeable, but they are so huge and heavy that they can only run in massive, expensive data centers. You can't put them in your pocket.

RightNow-Arabic-0.5B-Turbo is the "Goldilocks" solution: a tiny robot that speaks Arabic perfectly, designed specifically to fit in your pocket.

Here is how the authors built it, using simple analogies:

1. The Vocabulary Surgery (Giving the Robot a New Dictionary)

The original robot's dictionary was too small for Arabic. It had to chop Arabic words into tiny fragments, making it slow and inefficient.

The Fix: The team performed "vocabulary surgery." They took the robot's existing dictionary and injected 27,000 new Arabic words directly into it.
The Result: Instead of needing 2.18 tiny pieces to describe one Arabic word, the robot now only needs 1.80 pieces. It's like switching from a robot that spells out every letter of a word to one that recognizes whole words instantly. This makes the robot 17% faster when speaking Arabic.

2. The Training Camps (Teaching the Robot)

The team didn't just give the robot a new dictionary; they sent it through three intense training camps:

Camp 1: The Library (Continued Pretraining): They fed the robot 504 million Arabic sentences from Wikipedia. This was like giving the robot a crash course in reading Arabic newspapers and books so it could learn the rhythm and flow of the language.
Camp 2: The Classroom (Supervised Fine-Tuning): They taught the robot how to be a helpful assistant. They showed it 129,000 examples of questions and answers. Crucially, they told the robot, "Don't worry about the question part; just focus on learning how to write the answer." This made it much better at following instructions.
Camp 3: The Debate Club (Direct Preference Optimization): They showed the robot pairs of answers (one good, one bad) and asked it to pick the winner. However, because the robot is so small, this step didn't help much on its own. The "preference" data was a bit noisy, like trying to teach a toddler to judge fine art.

3. The "Weight Soup" (The Secret Sauce)

Here is the clever part. Since the "Debate Club" training didn't work perfectly, the team didn't just pick the final robot. Instead, they took three different versions of the robot (the one after the Library, the one after the Classroom, and the one after the Debate Club) and blended them together.

The Analogy: Imagine making a soup. You take a cup of the "Library" robot, a cup of the "Classroom" robot, and half a cup of the "Debate Club" robot, and mix them.
The Result: This "soup" created a final robot that was smarter and more balanced than any single version on its own.

4. The Final Product: Tiny but Mighty

The result is a model with 518 million parameters (tiny compared to the 7B or 9B giants).

Size: When compressed (quantized) to fit on a phone, it takes up only 398 MB of space. That's smaller than a high-definition movie.
Speed: On a powerful computer chip, it can generate text at 635 words per second. That's like a human typing at lightning speed.
Performance:
- It beats other tiny robots (like the original Qwen) at understanding Arabic.
- It ties with a robot three times its size (Falcon-H1-1.5B) on common sense reasoning tasks.
- It recovers 67% of the intelligence of the massive 9B robot (SILMA), despite being 18 times smaller.

What It Can't Do (The Limitations)

The paper is honest about what this tiny robot cannot do:

Knowledge Ceiling: It doesn't know as many facts as the giant robots. If you ask it complex trivia questions (like a hard quiz), it will lose to the 7B or 9B models. It's a specialist in language, not a encyclopedia of facts.
Dialects: It speaks "Modern Standard Arabic" (the formal language used in news and books). If you speak to it in Egyptian, Gulf, or Levantine dialects, it will understand you but reply in formal Arabic. It hasn't learned the street slang yet.

The Bottom Line

The authors built the smallest open-source, Arabic-specialized robot that actually works well. They proved you don't need a supercomputer to run a fluent Arabic assistant; you just need the right vocabulary, the right training mix, and a little bit of "soup" magic. All the code, weights, and recipes are now open for anyone to use.

Technical Summary: RightNow-Arabic-0.5B-Turbo

Problem Statement

The landscape of open Arabic Large Language Models (LLMs) is currently polarized. On one end, sub-1B multilingual models (e.g., Qwen2.5-0.5B, Falcon-H1-0.5B) treat Arabic as a secondary language, lacking specialized vocabulary or training data, resulting in high token fertility (inefficient encoding) and lower performance. On the other end, Arabic-specialized models (e.g., Jais, AceGPT, ALLaM, SILMA) range from 7B to 70B parameters. While these achieve strong benchmark scores, they require 16–140 GB of memory, rendering them inaccessible for edge deployment on phones, laptops, or embedded devices.

A previous attempt to bridge this gap, Kuwain-1.5B, proposed vocabulary injection but never released its weights, leaving a void for a deployable, sub-1B Arabic-specialized decoder. This paper addresses the need for a model that balances Arabic specialization with the strict memory and compute constraints of edge devices.

Methodology

The authors present RightNow-Arabic-0.5B-Turbo, a 518M-parameter decoder LLM built upon the Qwen2.5-0.5B base. The training pipeline consists of five distinct stages, orchestrated to maximize performance within the model's scale ceiling:

Vocabulary Injection (Tokenizer Surgery):
- The base Qwen2.5-0.5B tokenizer (151,665 tokens) was expanded by adding 27,032 new Arabic tokens.
- Initialization: New token embeddings were initialized using mean-subtoken initialization (averaging the embeddings of the new token's sub-piece decomposition from the original vocabulary), a technique formalized by WECHSEL and adopted by Kuwain and ALLaM.
- Result: This reduced Arabic token fertility from 2.18 to 1.80 tokens per word (a 17.3% reduction), directly lowering inference costs.
Continued Pretraining:
- The model was pretrained on 504M tokens of Arabic Wikipedia.
- Infrastructure: Training utilized 8×H100 GPUs with FSDP (Hybrid Sharding Zero2), FlashAttention varlen (for packing sequences without padding), and Liger fused kernels (for memory-efficient RMSNorm, RoPE, SwiGLU, and cross-entropy).
- Optimization: Fused AdamW with a peak learning rate of $2 \times 10^{-4}$ , cosine decay, and a total of 10.5B training tokens.
Supervised Fine-Tuning (SFT):
- The model was fine-tuned on 129,116 instruction pairs derived from five merged Arabic datasets.
- Loss Masking: Crucially, loss was computed only on the assistant response tokens, masking the prompt to concentrate the learning signal on the generation distribution.
Direct Preference Optimization (DPO):
- The SFT checkpoint underwent DPO on 6,750 Arabic preference pairs.
- Observation: The authors note that the preference signal was weak at this scale, with training loss remaining near $\ln 2$ and reward margins near zero, suggesting the dataset size or quality (machine-translated) was insufficient to drive significant improvement on its own.
Weight Soup Merging:
- To mitigate the weak DPO signal and recover capabilities, the authors merged three checkpoints: the DPO checkpoint, the SFT checkpoint, and the continued pretraining checkpoint.
- Configuration: A linear "soup" with weights of 50% DPO, 25% SFT, and 25% Pretraining yielded the best results, improving mean accuracy by 0.44 absolute points over the DPO checkpoint alone.
Edge Deployment:
- The final model was exported to llama.cpp GGUF format.
- Quantization: Four levels were produced (f16, q8_0, q5_k_m, q4_k_m). Due to vocabulary expansion misalignment with k-quant tile sizes, the effective bits-per-weight for q4_k_m and q5_k_m are 6.45 and 6.79 respectively, rather than the nominal 4 and 5.

Key Contributions

Smallest Open Arabic-Specialized LLM: The release of a 518M parameter model with publicly available weights, the first of its kind on HuggingFace.
Reproducible Pipeline: A complete, open-source pipeline (5,555 lines of Python across 25 scripts) detailing vocabulary injection, data loading (avoiding HuggingFace Hub stalls via memmap), and training.
Tokenizer Efficiency: A demonstrated 17.3% reduction in Arabic token fertility, translating to faster inference for the same semantic content.
Benchmarking: Head-to-head evaluation against six competing models on three Arabic tasks (COPA-ar, Arabic HellaSwag, ArabicMMLU), establishing a new baseline for sub-1B models.
Ablation on Merging: A systematic evaluation of seven weight-soup variants, identifying that averaging checkpoints improves generalization over relying solely on the DPO endpoint.
Edge Performance: Demonstration of the model running at 635 tokens/s on a single H100 (batch size 1) in 4-bit quantization, occupying only 398 MB on disk.

Experimental Results

Evaluated using lm-evaluation-harness (v0.4.11):

Same-Class Performance: RightNow-Arabic-0.5B-Turbo outperformed all other open sub-1B models (Qwen2.5-0.5B and Falcon-H1-0.5B) on COPA-ar (+4.5% and +13.5% respectively) and HellaSwag-ar. It achieved a mean accuracy of 35.9%, beating the multilingual baselines.
Scaling Comparison:
- The model recovers 67.1% of the mean accuracy of SILMA-9B (a 9B parameter model) while using only 5.8% of the parameters.
- It ties Falcon-H1-1.5B (1.5B parameters) on the COPA-ar benchmark (58.4%) despite having one-third the parameters.
- As expected, a significant gap remains on knowledge-intensive tasks (ArabicMMLU), where the 0.5B model trails 7B+ models by 12–30 points, confirming a "knowledge ceiling" dictated by parameter count.
Inference Speed: Using llama.cpp on an H100, the 4-bit quantized model achieves 635 tokens/s at batch size 1, an 8x speedup over standard HuggingFace generate() implementations due to optimized CUDA graphs and C++ sampling loops.

Significance and Claims

The paper claims that RightNow-Arabic-0.5B-Turbo is the smallest open Arabic-specialized decoder LLM released to date. Its significance lies in proving that careful orchestration of existing techniques—specifically vocabulary injection, response-only loss masking, and weight soup merging—can produce a model that is viable for edge deployment without requiring new training paradigms.

The authors are modest about the model's limitations, explicitly stating:

Knowledge Ceiling: The model cannot close the gap on knowledge-heavy benchmarks (like MMLU) against 7B+ models; parameter count is the limiting factor for world knowledge.
Dialect Limitations: The model is trained on Modern Standard Arabic (MSA) and performs poorly on dialects (Egyptian, Gulf, Levantine).
DPO Weakness: The DPO stage provided minimal signal at this scale, suggesting that preference tuning may require larger datasets or stronger reference models to be effective for sub-1B models.

The work targets the specific niche of users who need Arabic specialization on devices with limited memory (phones, edge CPUs) where 14GB+ models are infeasible, offering a practical, open alternative to the current "all-or-nothing" landscape.