Technical Summary: TFD – A Comprehensive Structured Tibetan Foundation Dataset

1. Problem Statement

Despite the remarkable success of Large Language Models (LLMs) in high-resource languages, progress in Tibetan remains severely constrained. The primary limitation is not merely the scarcity of raw Tibetan text, but the absence of a structurally complete data ecosystem that covers the entire modern LLM development pipeline. Existing Tibetan resources are typically small-scale, domain-specific, noisy, and lack standardized processing. Crucially, no prior resource jointly supports the full spectrum of modern training stages: pre-training, instruction tuning, safety alignment, preference optimization, and reasoning supervision. Consequently, current Tibetan LLMs exhibit limited generalization, weak discourse modeling, and unstable generation.

2. Methodology: The Tibetan Foundation Dataset (TFD)

To address this structural gap, the authors introduce TFD, the first structured, large-scale, and expert-curated dataset designed to cover the complete LLM development pipeline for Tibetan. TFD comprises two primary sub-datasets:

2.1 TIBSTC (General Supervised Fine-Tuning)

TIBSTC is a multi-component collection containing over 11 billion tokens across five categories:

• Corpus: A heterogeneous pre-training corpus (122M+ instances) covering literature, law, religion, medicine, and daily communication. It includes traditional grammar and syntax, preserving cultural distinctiveness.
• Alpaca-Ti: A dataset for supervised instruction tuning (42k instances), constructed by translating and locally adapting the English Alpaca dataset. It underwent a rigorous two-stage expert alignment protocol involving dual-model drafting (Google Translate/Claude-3.5-Sonnet) followed by human evaluation for semantic fidelity and cultural alignment.
• Safety-Prompts-Ti: A safety alignment dataset (38k instances) covering instruction attacks (e.g., goal hijacking, prompt leaking) and safety scenarios (e.g., physical harm, ethics).
• CValues-Ti: A small-scale preference dataset (2k instances) focusing on safety-oriented response categorization (Safe & Responsibility, Safe, Unsafe).
• hh-rlhf-Ti: A preference optimization dataset (47k instances) adapted from HH-RLHF, reorganized to fit Tibetan linguistic patterns for improving helpfulness and harmlessness.

2.2 TIBSTC-CoT (Chain-of-Thought Reasoning)

This is the first large-scale native Tibetan chain-of-thought dataset. It was constructed via a multi-stage, multi-model pipeline involving:

1. Question Formulation: An agent generates diverse, contextually relevant questions across five domains (Science, Life Sciences, Humanities, Social Sciences, Comprehensive).
2. CoT and Answer Generation: A generator model produces intermediate reasoning steps and final answers.
3. Automatic Assessment: An evaluator model scores samples on coherence, logic, and cultural appropriateness.
4. Manual Verification: Human experts validate the retained samples for factual consistency and cultural norms.

2.3 Data Processing Pipeline

The authors emphasize a rigorous curation process. Unlike simple machine translation, the pipeline involves:

• Dual-Model Drafting: Generating drafts with multiple LLMs.
• Human Evaluation: Independent scoring by Tibetan specialists on semantic fidelity, fluency, domain knowledge, and cultural alignment.
• Expert Adjudication: Two rounds of review to raise the expert approval rate from ~28% (initial draft) to 100%.
• Linguistic Normalization: Ensuring orthographic consistency and correcting unnatural syntax.

3. Model Training: The Sun-Shine Family

To validate TFD, the authors trained the Sun-Shine family of reference LLMs:

• Sun-Shine 1.0: Built on the Llama-3.1 (8B) backbone. It follows a three-stage pipeline: Pre-training on the TIBSTC Corpus, Supervised Fine-Tuning (SFT) on Alpaca-Ti and Safety-Prompts-Ti, and Direct Preference Optimization (DPO) on CValues-Ti and hh-rlhf-Ti.
• Sun-Shine 2.0 (Sunshine-Thinking): Built on the Qwen-3 backbone (1.7B and 8B variants). It is trained via SFT using TIBSTC-CoT to explicitly learn multi-step reasoning.

4. Key Results

The authors evaluated the Sun-Shine models against strong baselines (including GPT-4o, DeepSeek-R1, and various Qwen/Llama variants) on the TLUE benchmark and downstream tasks.

4.1 Understanding and Safety (TLUE)

• Performance: Sun-Shine 1.0 (8B) outperformed significantly larger models (e.g., Llama-3.1-405B, DeepSeek-R1) in Response Rate (RR) and Accuracy (ACC) across China-specific, STEM, and other categories on Ti-MMLU.
• Safety: Sun-Shine 1.0 excelled in safety benchmarks (Ti-SafetyBench), particularly in categories like Illegal Activities and Offensiveness, attributed to targeted supervision from Safety-Prompts-Ti.
• Reasoning: Sun-Shine 2.0 (Sunshine-Thinking) consistently outperformed baselines in multi-step reasoning tasks. Notably, the 1.7B variant surpassed larger models (e.g., GPT-4.1, Qwen-3-8B) in Social Sciences, demonstrating that structured supervision (CoT) yields gains independent of parameter scale.

4.2 Generation and Translation

• Translation: Sun-Shine 1.0 achieved the highest BLEU-4 and ROUGE scores in Tibetan-Chinese/English translation tasks, outperforming GPT-4o and DeepSeek-R1.
• Cultural Alignment: In qualitative evaluations (classical Chinese generation, ancient poetry), Sun-Shine 1.0 maintained a restrained, culturally aligned style, whereas larger general-purpose models produced more modernized or embellished outputs.

4.3 Ablation Studies

Token-matched ablation studies confirmed that performance gains in reasoning tasks were driven by the supervision structure (CoT data) rather than increased token volume, as CoT-SFT outperformed standard SFT and DeCoT-SFT (without reasoning steps) despite identical token budgets.

5. Significance and Claims

The paper claims that advancing low-resource language modeling requires not just scale, but a structurally complete data ecosystem.

• First of its Kind: TFD is the first resource to jointly support pre-training, instruction tuning, safety alignment, preference optimization, and reasoning supervision for Tibetan.
• Efficiency: The results demonstrate that an 8B model trained on a structured, curated dataset can compete with or surpass models an order of magnitude larger, highlighting the efficacy of high-quality data over sheer parameter count.
• Reproducibility: By releasing TFD and the Sun-Shine models, the authors aim to facilitate reproducible research and the development of robust, culturally aligned Tibetan LLMs.

The authors acknowledge limitations, including underrepresentation of specialized dialects, potential residual biases from LLM-assisted generation in TIBSTC-CoT, and the current lack of multimodal evaluation. They position this work as a foundational step, leaving further architectural optimization and training strategy exploration to future research.