SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

1. Problem Statement

The paper addresses the significant underservice of Kazakh, a low-resource, agglutinative Turkic language spoken by over 22 million people, by current Large Language Models (LLMs). The authors identify two primary inefficiencies in applying general-purpose multilingual models (e.g., Llama 3, Gemma, Qwen) to Kazakh:

• Tokenizer Inefficiency: Multilingual tokenizers, trained on diverse languages, fragment Kazakh words into an excessive number of subword tokens due to its rich morphology (extensive inflectional and derivational suffixes). This high "fertility" (tokens per word) reduces the effective context window and increases inference costs.
• Capacity Misallocation: In multilingual models, the vast majority of parameters encode knowledge of high-resource languages (like English), leaving a poor ratio of useful capacity to total model size for Kazakh-specific tasks.

The authors hypothesize that small, dedicated monolingual models trained from scratch with a language-specific tokenizer can achieve competitive performance against much larger multilingual models at a fraction of the computational cost.

2. Methodology

The authors developed SozKZ, a family of causal language models based on the Llama architecture, trained entirely from scratch.

Data Pipeline

• Corpus Construction: Aggregated 18 sources (including Kazakh Wikipedia, CulturaX, mC4, and MADLAD-400), resulting in 28.4 million raw documents.
• Cleaning: Applied a rigorous 9-stage cleaning pipeline (NFC normalization, control character removal, URL/HTML filtering, script-based language identification, and deduplication).
• Final Dataset: 13.7 million documents (~9 billion tokens) remained after filtering, representing one of the largest publicly available monolingual Kazakh corpora.

Tokenizer Design

• Architecture: A dedicated ByteLevel BPE tokenizer trained exclusively on the cleaned Kazakh corpus.
• Vocabulary: 50,257 tokens.
• Advantage: Designed to minimize fertility (tokens per word) for agglutinative structures, offering a 2–3× efficiency gain over multilingual tokenizers.

Model Architecture & Training

• Scales: Four model sizes were trained: 50M, 150M, 300M, and 600M parameters.
• Architecture: Follows the Llama-For-CausalLM design with SwiGLU activations, Rotary Position Embeddings (RoPE), RMSNorm, and tied input-output embeddings.
• Training Regime:
  • Trained for one epoch on ~9B tokens.
  • Data-to-Parameter Ratio: ~15:1 for the 600M model (slightly below the Chinchilla-optimal 20:1, reflecting data scarcity).
  • Hardware: 50M/150M on 2× RTX 4090; 600M on 8× H100 GPUs.
  • Optimization: AdamW optimizer with cosine learning rate scheduling.

3. Key Contributions

1. End-to-End Pipeline: A reproducible pipeline for training Kazakh LLMs from scratch, including data collection, cleaning, and pre-tokenization.
2. Specialized Tokenizer: A 50K vocabulary ByteLevel BPE tokenizer that significantly improves tokenization efficiency for Kazakh compared to general multilingual alternatives.
3. Scaling Analysis: Empirical evidence of scaling laws for Kazakh, demonstrating consistent performance gains from 50M to 600M parameters.
4. Open Release: All models, the tokenizer, and training code are released under open licenses to foster Kazakh NLP research.

4. Experimental Results

The models were evaluated in a zero-shot setting on three Kazakh benchmarks against five multilingual baselines (Qwen, Llama 3, Gemma) ranging from 0.5B to 3B parameters.

• Multiple-Choice Cultural QA (kk-socio-cultural-bench-mc):

  • SozKZ-600M achieved 30.3% accuracy.
  • This approaches the performance of Llama-3.2-1B (32.0%) and Qwen-0.5B (31.5%), despite SozKZ-600M having significantly fewer parameters than the 3B models and being trained on monolingual data only.
  • Scaling was consistent: Accuracy rose from 22.8% (50M) to 30.3% (600M).

• Reading Comprehension (Belebele):

  • Performance was more uniform across models, with SozKZ-600M achieving 27.0%, comparable to Llama-3.2-1B (26.7%).
  • The authors note this benchmark may be near the "floor" for base models without instruction tuning.

• Topic Classification (SIB-200):

  • SozKZ models demonstrated a clear advantage. SozKZ-600M achieved 25.5% accuracy.
  • Crucially, even the smallest SozKZ-50M model (25.5%) outperformed all multilingual models up to 2B parameters (e.g., Gemma-2B: 20.1%, Qwen-1.5B: 11.8%).
  • This confirms the hypothesis that a dedicated tokenizer preserves morphological word boundaries, providing superior lexical signals for classification tasks.

5. Significance and Conclusion

The paper demonstrates that specialization yields efficiency. By training small models from scratch on a dedicated corpus with a language-appropriate tokenizer, the SozKZ family achieves competitive or superior performance to multilingual models that are 2–5× larger.

• Efficiency: The 600M SozKZ model rivals 1B–3B multilingual models, offering a viable path for low-resource language technology with lower computational costs.
• Tokenizer Impact: The results strongly validate that for agglutinative languages, the choice of tokenizer is as critical as model architecture.
• Future Directions: The authors suggest that further scaling (1–3B range) and instruction tuning could close the remaining gaps in knowledge-intensive tasks and reading comprehension.

In summary, SozKZ provides a blueprint for developing efficient, high-performance language models for underserved languages, challenging the notion that massive multilingual models are the only solution for low-resource NLP.