Technical Summary: Jais 2 – A Family of Arabic-Centric Open Large Language Models

Problem Statement

Despite rapid advancements in multilingual Large Language Models (LLMs), the field remains heavily biased toward English and a limited set of high-resource languages. Arabic, characterized by its complex morphology, diglossia (the coexistence of Modern Standard Arabic and diverse regional dialects), and rich cultural context, is significantly underrepresented in global training corpora. Existing general-purpose models (e.g., Llama 3, Gemma, Qwen) often exhibit partial proficiency in Arabic, struggling particularly with dialectal variations, culturally nuanced reasoning, and specific domains such as Islamic jurisprudence, poetry, and regional cuisine. Furthermore, many existing Arabic-centric initiatives rely on adapting multilingual backbones or are limited to narrow linguistic domains, lacking the scale and cultural grounding necessary for robust, real-world deployment across the Arab world.

Methodology

Model Architecture and Training Infrastructure

Jais 2 is a decoder-only Transformer family released in two variants: 8B and 70B parameters. Both models are trained from scratch.

• Architecture: The models utilize an expanded feedforward filter ratio (8× the hidden size), ReLU2 activations to induce sparsity, and Rotary Position Embeddings (RoPE) for superior context extension. Unlike earlier Jais versions which used ALiBi, Jais 2 adopts RoPE. The architecture employs Maximal Update Parameterization (µP) to enable zero-shot transfer of optimal hyperparameters from small proxy models to large-scale variants.
• Tokenization: A custom Byte-Pair Encoding (BPE) tokenizer with a vocabulary size of 150,272 was developed, optimized specifically for Arabic, English, and code, while preserving space-prefixed tokens crucial for formatting and semantics.
• Infrastructure: Training was conducted on Condor Galaxy 1 and 2, Cerebras Wafer-Scale Clusters comprising up to 64 CS-2 systems. This architecture enabled near-perfect linear scaling, allowing the training of the 70B model with a substantially smaller token budget than comparable models.

Data Strategy and Pretraining

Jais 2 employs a two-stage training curriculum:

1. Stage 1 (Pretraining): The model is trained on a mixture of 600B+ Arabic tokens and 1.6T total tokens. The data mix includes:
   • Arabic Data: Curated from diverse sources including web data, religious texts, poetry, literature, and scientific content. Crucially, this includes 17 dialectal variants (e.g., Moroccan Darija, Egyptian, Gulf, Levantine) and Arabizi (Latin-script Arabic).
   • English/Web Data: Selected from a comparative study of three major web corpora, where Corpus 3 was chosen for its superior cross-lingual transfer capabilities to Arabic.
   • Math and Code: Integrated to ensure reasoning and coding capabilities.
   • Mix Ratio: The final pretraining mix consists of 28% Arabic, 40% Web, 20% Math, 10% Code, and 2% Synthetic data.
2. Stage 2 (Annealing): A domain upsampling phase (10% of total FLOPs) increases the proportion of high-quality Arabic, math, and web data to address specific weaknesses identified after initial training.

Post-Training

The post-training pipeline consists of three distinct stages:

1. Continual Pretraining (CPT): Two epochs of training on new curated Arabic data and replayed original data to refine domain expertise.
2. Instruction Fine-Tuning (IFT): Three epochs using over 20 million instruction-completion pairs. The dataset includes:
   • General IFT: Standard Arabic and English instructions, including synthetic data for multi-turn dialogue and safety.
   • Dialectal IFT: Specific datasets for Moroccan Darija and Egyptian Arabic, covering translation, transliteration, and summarization.
   • Cultural Tasks: Specialized datasets for Arabic Poetry (generation and analysis), Islamic Question-Answering (with ethical disclaimers), Dream Interpretation, Arabic Cuisine, and Summarization.
3. Preference Alignment: The models are aligned using Direct Preference Optimization (DPO) on over 200k preference pairs (chosen vs. rejected responses) to ensure safety, helpfulness, and adherence to human values.

Safety Mechanisms

Safety is integrated through:

• Data Preparation: A unified taxonomy of offensive language (general, obscene, hate speech) was constructed from 30 datasets, resulting in 205k samples. These were enriched with explanatory IFT data to teach the model why content is offensive.
• SFT Safety: A synthetic dataset of 116k QA pairs covering 30 risk types (5 general, 25 regional) was generated to guide the model in handling sensitive political, religious, and social topics specific to the Arab world.

Key Contributions

1. Open-Weight Arabic-Centric Models: Release of Jais 2-8B and Jais 2-70B, trained from scratch. The 70B variant is among the largest open Arabic-centric models trained entirely from scratch.
2. Cultural and Dialectal Grounding: Unlike previous models focused primarily on Modern Standard Arabic (MSA), Jais 2 is explicitly trained on 17 dialects and culturally specific domains (poetry, cuisine, dream interpretation, Islamic QA), enabling deep cultural alignment.
3. Efficient Training: The use of a custom Arabic-centric vocabulary, µP, and Cerebras hardware allows the models to achieve strong performance with a significantly smaller token budget compared to comparable models.
4. Comprehensive Benchmarks: Introduction and evaluation on culturally grounded benchmarks including OALL2 (Open Arabic Leaderboard 2), AraGen, and custom benchmarks for cuisine, poetry, and dream interpretation.
5. Deployment: Release of the models on HuggingFace under a commercially permissive license, alongside a high-throughput chat application (up to 2,000 tokens/second) running on Cerebras hardware.

Evaluation Results

Jais 2 demonstrates leading performance among evaluated open models across multiple dimensions:

• Multiple Choice (OALL2): Jais 2-70B achieves a macro-average of 79.36%, outperforming Llama-3.3-70B (74.23%) and Qwen2.5-72B (71.20%). Jais 2-8B leads the ≤13B category with 72.40%.
• Generative Tasks (AraGen): Jais 2 models achieve the highest 3C3H scores (Correctness, Completeness, Conciseness, Helpfulness, Honesty, Harmlessness) in their respective size categories, with Jais 2-70B scoring 70.71%.
• Translation: Jais 2-70B sets new state-of-the-art scores for open models in translating between MSA, English, and various dialects (Egyptian, Tunisian, Levantine, Gulf, etc.), outperforming Llama-3.3-70B and Qwen2.5-72B in both directions.
• Dialect Identification: Jais 2-70B achieves the highest accuracy on MADAR and QADI benchmarks, significantly outperforming other 70B-scale models.
• Cultural Benchmarks:
  • Cuisine: Jais 2-70B achieves 79.02% accuracy on a 1,597-question benchmark, leading in Ingredient Analysis, Procedural Execution, and Geographic Provenance.
  • Poetry: Jais 2-70B leads in poetry generation and analysis tasks.
• English Capabilities: Despite its Arabic focus, Jais 2-70B remains competitive in English benchmarks (e.g., 86.09% on IFEval, 85.97% on GSM8K), demonstrating that strong Arabic specialization does not come at the expense of English performance.

Significance

The paper positions Jais 2 as a foundational step toward democratizing access to advanced AI for the Arabic-speaking world. By unifying scale, linguistic diversity (MSA and 17 dialects), cultural fidelity, and openness, Jais 2 provides a transparent and reproducible baseline for future research. The authors emphasize that the model is designed not only to bridge the linguistic gap in global AI but to support the next generation of Arabic-specific applications, research, and innovation, ensuring that regional AI development is built upon inclusive and culturally grounded foundations. The release of the models and the accompanying high-performance chat infrastructure aims to foster community research and practical deployment across the Arab world.