Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque

1. Problem Statement

Large Language Models (LLMs) remain predominantly English-centric, exhibiting significant performance degradation in low-resource languages (LRLs). While base models can be adapted via continued pretraining, instruction-tuning for LRLs faces specific challenges:

• Data Scarcity: High-quality, manually curated instruction datasets (prompt-response pairs) do not exist for most LRLs.
• Pipeline Limitations: The standard pipeline (Continued Pretraining $\rightarrow$ Instruction Tuning) often relies on translating English instructions or using synthetic data, but it is unclear if this is optimal.
• Evaluation Gaps: Automated benchmarks often fail to capture the nuances of user preference and linguistic fluency in LRLs, while human evaluation is rarely scaled for these languages.

The authors aim to identify the optimal adaptation strategy for Basque (a language isolate with ~1,000x less data than English) without relying on distillation from proprietary commercial models.

2. Methodology

The study employs a systematic, combinatorial experimental framework to test different adaptation strategies.

A. Resources & Components

• Backbone Models: Three base architectures were used:
  1. BASE_EN: Llama 3.1 8B/70B (English-only pretraining).
  2. BASE_EU: Llama 3.1 continued pretraining on Basque corpora (Latxa).
  3. INSTRUCT_EN: Llama 3.1 Instruct (English instruction-tuned).
• Training Data:
  • Corpus_EU: 4.3M high-quality Basque documents (~3.5B tokens).
  • Synthetic Instructions: Generated using the Magpie technique (sampling from the model's own user prompt header).
    • English Instructions: 1M samples generated from INSTRUCT_EN.
    • Basque Instructions: Translated from English using BASE_EU via few-shot prompting (chosen over NLLB for better handling of code/complex structures).
• Experimental Design: The authors tested $3 \times 2^3 = 24$ theoretical combinations of backbones and data (Corpus, English Instructions, Basque Instructions), excluding redundant configurations. This resulted in 18 distinct 8B models and 1 70B model.

B. Evaluation Framework

To overcome the limitations of static benchmarks, the authors implemented a dual-evaluation strategy:

1. Static Benchmarks: 27 benchmarks across Basque, English, and Spanish (Reading, Common Sense, Linguistics, Knowledge, Math/Reasoning).
2. Human Evaluation (Arena): A community-driven "Chatbot Arena" involving 1,680 participants who generated 12,890 preference annotations.
   • Participants rated pairs of models on Content Quality and Linguistic Quality (separately, as fluency is a major issue in LRLs).
   • Rankings were derived using the Bradley-Terry model.
3. Safety & Bias: Evaluated using adapted XSTest (safety) and BasqBBQ (bias) datasets.

3. Key Contributions

1. Systematic Strategy Discovery: A comprehensive analysis proving that starting from an instruction-tuned English backbone (INSTRUCT_EN) is superior to starting from a base model, even for low-resource languages.
2. Resource Efficiency: Demonstrated that synthetic instructions (generated by the model itself) are sufficient to create robust LRL models, eliminating the need for expensive manual data creation.
3. Largest LRL Human Evaluation: The largest human evaluation effort for a low-resource language to date (1,680 participants, 12k+ annotations).
4. Open Release:
   • Latxa-Instruct: A family of instruction-tuned models for Basque (8B and 70B parameters).
   • Datasets: Synthetic instruction datasets (English/Basque) and the first Basque preference dataset.
   • Code & Reproducibility: Full training code and evaluation scripts.

4. Key Results

A. Adaptation Strategies

• Backbone Choice: Starting from INSTRUCT_EN (an English instruction-tuned model) significantly outperformed starting from BASE_EN or BASE_EU. This challenges the standard "Pretrain $\rightarrow$ Instruct" pipeline, suggesting that instruction-following capabilities transfer better than language capabilities.
• Corpus Importance: Exposure to the target language corpus (Corpus_EU) is essential. Models without Basque text showed severe degradation, regardless of instruction data.
• Instruction Language: While English-only instructions worked, bilingual instructions (English + Basque) yielded the most robust models across both benchmarks and human preferences.
• Curriculum Learning: No significant benefit was found in separating language acquisition (pretraining) from instruction tuning; simultaneous training or starting from an instructed model was more effective.

B. Performance

• Benchmark Performance: The best 8B model (INSTRUCT_EN + Corpus_EU + Instructions_EN+EU) achieved a 64.20% average accuracy on Basque benchmarks.
• Scaling: Scaling to 70B parameters yielded a 76.51% average accuracy, approaching the performance of proprietary models.
• Arena Rankings:
  • The 70B Basque model ranked 3rd globally in the arena (behind GPT-4o and Claude 3.5 Sonnet), outperforming other open models.
  • It achieved par with commercial models in Linguistic Quality but trailed slightly in Content Quality.
  • Notably, the model outperformed GPT-4o on BertaQA Local (Basque cultural knowledge) and MGSM (Math).

C. Safety & Bias

• Safety alignment (low Violation Rates) was successfully transferred from the English backbone to Basque.
• The 70B model showed lower False Refusal Rates (FRR) and better bias handling than smaller variants.

5. Significance

• Paradigm Shift: The paper suggests that for LRLs, the most effective path is not to pretrain a base model from scratch and then instruct it, but to adapt an existing high-quality instruction-tuned model (even if English-only) using target language corpora and synthetic instructions.
• Scalability: The methodology is applicable to any language with similar resource profiles (e.g., ~50th in Common Crawl presence), offering a blueprint for democratizing LLM access.
• Community Impact: By releasing the first large-scale Basque instruction and preference datasets, the authors enable future research into preference alignment and further model refinement for the Basque-speaking community.

In conclusion, the study demonstrates that with the right combination of a strong instructed backbone, target language corpora, and synthetic bilingual instructions, open-source models can achieve near-frontier performance in low-resource languages without relying on proprietary distillation.