Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

Technical Summary: Vikhr – The Family of Open-Source Instruction-Tuned Large Language Models for Russian

Problem Statement
The paper addresses the significant challenges faced by Large Language Models (LLMs) when generating text in non-English languages, specifically Russian. While English-oriented LLMs possess some multilingual capabilities, their performance in other languages is limited by poor generation quality, lack of culture-specific knowledge, and computational inefficiency. This inefficiency arises from byte-pair tokenization algorithms, which split infrequent non-English words into multiple tokens, thereby increasing processing steps, reducing the effective context window, and degrading overall performance. Existing solutions often rely on LoRA adapters atop English models, which improve generation quality but fail to address tokenization inefficiencies. Furthermore, while proprietary Russian LLMs exist, the lack of open-source, instruction-tuned models with white-box access hinders reproducible research.

Methodology
The authors developed a comprehensive pipeline to adapt English-oriented pre-trained models (specifically Mistral 7B) into efficient bilingual LLMs for Russian and English. The pipeline consists of three main stages:

1. Vocabulary Adaptation: To mitigate tokenization inefficiency, the authors rebuilt the tokenizer using a language-specific corpus (RuLM). They trained a SentencePiece tokenizer with a 40,000-token vocabulary. This resulted in significantly higher tokenization density for Russian compared to the original Mistral tokenizer, reducing the number of tokens required per word.
2. Continued Pre-training: The model underwent continued pre-training on a curated corpus of 11 billion Russian tokens. The data sources included Russian Wikipedia, news articles, scientific papers, and top posts from Habr. The corpus was deduplicated using MIHash and filtered for quality using a fine-tuned RuBERT-tiny model. To prevent "catastrophic forgetting" of the base model’s reasoning capabilities, the authors employed loss regularization with a KL penalty between the probability distributions of the adapted model and the reference English model.
3. Instruction Tuning: The model was fine-tuned on an expanded set of instruction-output pairs. This dataset combined the existing Saiga dataset (100k instances) with newly translated datasets: Veles (30k instances, translated from OpenHermes) and ruFLAN (500k instances, translated from FLAN). Additionally, the English Nectar dataset (50k instances) was included to maintain English proficiency. Low-quality pairs were filtered out using a multilingual-e5-large reward model. Unlike previous approaches, all model weights were updated during this phase, rather than using LoRA adapters.

Key Contributions

• Adaptation Pipeline: Development of a pipeline for adapting English LLMs to other languages, incorporating vocabulary adaptation, continued pre-training with regularization, and full-weight instruction tuning.
• Vikhr Model: Construction of Vikhr, an open-source, instruction-following bilingual LLM for Russian and English, featuring an efficient tokenizer and high computational efficiency.
• Dataset Expansion: Creation of expanded datasets for continued pre-training and instruction tuning, including the translation and cleaning of English instruction sets for Russian use.
• Evaluation Benchmarks: Construction of Ru-MMLU and Ru-MMLU-pro benchmarks by translating the English MMLU and MMLU-pro benchmarks.
• Ablation Studies: Empirical validation of the pipeline components, confirming the effectiveness of vocabulary adaptation, data filtration, and loss regularization.

Results

• Tokenization Efficiency: The Vikhr tokenizer demonstrated superior efficiency for Russian text compared to Mistral, Llama3, and ruGPT tokenizers, as shown by higher tokenization density.
• Generation Quality: Qualitative analysis revealed that Vikhr generates grammatically coherent and culturally appropriate Russian text, whereas the base Mistral model often injected English words, struggled with grammar, and misunderstood Russian word meanings.
• Ablation Findings:
  • Vocabulary Size: A vocabulary size of 40k tokens provided the optimal balance between perplexity and performance on the ruXNLI task.
  • Data Filtration: Pre-training on the filtered corpus (11B tokens) yielded lower perplexity and higher scores on Ru-MMLU-pro and ruXNLI compared to the unfiltered corpus (17B tokens), highlighting the importance of data quality.
  • Regularization: KL loss regularization during continued pre-training reduced perplexity and significantly improved performance on ruXNLI, mitigating catastrophic forgetting.
  • Instruction Sets: Fine-tuning on the authors' combined instruction set resulted in higher performance on Ru-MMLU-pro compared to fine-tuning on Saiga or translated Nectar alone.

Significance and Claims
The paper claims that Vikhr fills a critical gap in the availability of open-source, instruction-tuned LLMs for Russian that are both efficient and effective. By adapting the tokenizer and performing full-model continued pre-training and instruction tuning, Vikhr achieves high generation quality and computational efficiency. The authors emphasize that the developed cross-lingual adaptation pipeline and the published models aim to foster research on LLMs and enhance the diversity of languages in research agendas. They note that while they did not implement RLHF or DPO due to resource constraints, the model partially inherits alignment from the ChatGPT-derived training data. The authors plan to release improved versions and extend the pipeline to low-resource languages such as Belarusian, Serbian, and Kazakh.