Technical Summary: Soro – A Lightweight Foundation Model and Chatbot for Tajik

Problem Statement

Large Language Models (LLMs) exhibit strong performance in high-resource languages like English and Chinese but remain heavily skewed against low-resource languages. Tajik, an Indo-Iranian language written in Cyrillic and spoken primarily in Tajikistan, suffers from a sparse digital corpus, scarce annotated resources, and minimal representation in mainstream multilingual training data. This scarcity restricts access to modern AI tools in critical sectors such as education, administration, and cultural preservation. Furthermore, deploying large models in Tajikistan faces practical constraints: limited compute infrastructure, scarce high-end GPUs in educational institutions, and intermittent internet connectivity in rural regions. Existing multilingual models often fail to provide robust linguistic competence or culturally grounded factual knowledge in Tajik, and standard evaluation benchmarks for the language are virtually non-existent.

Methodology

The authors present Soro, a family of Tajik-specialized conversational LLMs (27B and 12B parameters) built upon open-weight Gemma 3 checkpoints. The development process follows a three-stage pipeline designed for parameter efficiency and deployment feasibility:

1. Base Model Selection: The authors evaluated several foundation models (Gemma 3, Qwen 3, Llama 3, GPT-OSS) based on deployment feasibility (8B–30B range), prior multilingual performance, and tokenization efficiency. Gemma 3 was selected due to its superior Tajik tokenization efficiency (lowest fertility rate) and strong baseline performance, which preserves effective context windows for Tajik text.
2. Data Curation:
   • Continual Pretraining: A curated, Tajik-only corpus of 1.9 billion tokens was assembled. Sources included the Tajik subset of FineWeb2, translated FineWeb-Edu (for STEM and academic grounding), FinePDFs (for formal documents), and a unique subset of national secondary school textbooks (grades 5–11) covering history, literature, and sciences. These textbooks were processed via manual transcription and high-fidelity OCR.
   • Instruction Tuning: A synthetic dataset of 40,000 examples was generated using Gemini 2.5 Flash, conditioned on the curated educational materials. The data follows a "teacher-like" style, emphasizing informative, accessible, and pedagogically sound responses in Tajik.
3. Training and Adaptation:
   • Continual Pretraining: Performed using LoRA (Low-Rank Adaptation) on the 12B and 27B Gemma 3 bases. The 12B model used a LoRA rank of 128, and the 27B model used 64, with full training of the embedding layer to refine Tajik-specific subword patterns.
   • Instruction Tuning: Applied to the pre-trained checkpoints using a reduced LoRA rank (32) to prevent overfitting on the smaller instruction dataset.
   • Linear Merging: To recover general-purpose capabilities potentially lost during specialization, the final Soro checkpoints were linearly merged with the original Gemma 3 instruction-tuned (IT) weights. An optimal ratio of 80% Soro / 20% Gemma 3-IT was identified, balancing Tajik-specific knowledge with general reasoning.
4. Quantization for Edge Deployment: To address hardware constraints, the merged models were quantized using Dynamic FP8 and GPTQ INT4. These methods reduced memory footprints by approximately 50% and 75% respectively, enabling deployment on consumer-grade GPUs (e.g., NVIDIA RTX 3060/4060) and edge devices.

Key Contributions

• Soro Model Family: The release of the first Tajik-centric LLMs developed within Tajikistan, specifically optimized for low-resource, low-connectivity environments.
• Tajik Benchmark Suite: The creation and open-sourcing of a comprehensive evaluation suite to address the lack of Tajik benchmarks. This includes:
  • Tajik-FactQA: University entrance exam questions on history, culture, and geography.
  • Tajik History & Literature: Curriculum-aligned questions from grades 5–11.
  • TajLib: A linguistic competence benchmark covering phonetics, grammar, and orthography.
  • Tajik MMLU & Tajik Curated: Translations and adaptations of general knowledge benchmarks.
• Education Sector Pilot: A structured deployment of Soro across 100 schools in five regions of Tajikistan (Dushanbe, Khujand, Vahdat, Kulob, Khorog), integrated into the national "Introduction to AI" curriculum.
• Quantization Strategy: Demonstration that aggressive quantization (FP8/INT4) preserves Tajik-language gains while enabling edge deployment on hardware accessible to schools.

Results

• Benchmark Performance: Across the Tajik-specific benchmark suite, Soro (27B) and Soro Lite (12B) substantially outperformed same-size Gemma 3 baselines and other open-source models (e.g., Qwen, Llama).
  • Soro achieved an average accuracy improvement of 6 percentage points over its Gemma 3 base.
  • On TajLib (linguistic competence), gains exceeded 10 percentage points.
  • Soro Lite (12B) surpassed larger models like Qwen3-14B, and Soro (27B) outperformed Qwen3-32B despite having fewer parameters.
• General Capability Retention: Performance on the English MMLU benchmark remained stable, indicating that Tajik-focused adaptation did not degrade general reasoning or English knowledge.
• Quantization Impact:
  • FP8: Reduced memory by ~50% with negligible accuracy loss (e.g., 27B model accuracy remained effectively identical to full precision).
  • INT4: Achieved 4× memory compression (13.5 GB for the 27B model) while retaining the majority of performance gains over baselines.
• Pilot Feedback: In the initial deployment across 100 schools, teachers reported high satisfaction with Tajik language quality (4.47/5 rating) and lesson preparation utility. Students showed increased engagement, though some latency issues and occasional factual errors on hyper-local topics were noted.

Significance

The paper positions Soro as a concrete example of how Type II Partial AI Sovereignty can be achieved: adapting open-weight models to encode local knowledge while maintaining feasible local deployment. The work demonstrates that focused, resource-conscious adaptation of strong foundation models can produce high-quality, deployable language technology for severely under-resourced languages.

The authors emphasize that the methodology—combining curated monolingual pretraining, synthetic instruction tuning, linear merging, and aggressive quantization—is directly transferable to other low-resource languages facing similar infrastructure constraints. By providing both the models and the first rigorous evaluation suite for Tajik, the work aims to serve as a practical blueprint for expanding AI access in education and public sectors across Tajikistan and similar regions, moving beyond theoretical capability to real-world utility in schools and government offices.