Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models

Technical Summary: Maistros – A Greek Large Language Model Adapted Through Knowledge Distillation

Problem Statement
Despite the rapid advancement of Large Language Models (LLMs) and the emergence of Large Reasoning Models (LRMs) capable of complex multi-step reasoning, significant disparities remain in performance between high-resource and under-resourced languages. Modern Greek, characterized by its distinct alphabet, rich morphology, and syntactic variability, remains an under-resourced language in the NLP landscape. Existing Greek-capable models often rely on limited training data, resulting in suboptimal performance on Question Answering (QA) tasks, particularly those requiring cultural or domain-specific knowledge. Furthermore, while recent LRMs improve accuracy through explicit reasoning traces, their massive parameter counts (hundreds of billions) and high inference latency render them impractical for deployment in conventional computing environments. A critical gap exists in the availability of high-quality, human-curated Greek QA datasets suitable for training, as most existing resources are designed solely for evaluation.

Methodology
To address these challenges, the authors propose a pipeline centered on knowledge distillation from a powerful LRM to a compact, open-weights model, supported by a newly curated dataset.

1. Dataset Creation (CulturaQA): The authors introduced CulturaQA, a synthetic and human-curated dataset comprising 2,700 Greek question-answer pairs. The generation process involved:

   • Selecting 180 Greek keyphrases across 11 cultural domains (e.g., history, politics, economy, mythology).
   • Generating 15 questions per keyphrase using GPT-5, followed by the model answering its own questions.
   • Rigorous human-in-the-loop curation to correct linguistic errors (grammar, spelling, polytonic issues), remove hallucinations, fix factual inaccuracies, and eliminate non-Greek artifacts.
   • Stratified splitting into training (2,000), validation (200), and testing (500) sets.

2. Model Development (Maistros 8B):

   • Base Model: The authors selected Ministral 3 8B, a general-purpose open-weights LLM, as the foundation.
   • Fine-Tuning Strategy: They employed Supervised Fine-Tuning (SFT) using Low-Rank Adaptation (LoRA) to distill knowledge from CulturaQA into the base model.
   • Technical Configuration: Training utilized 4-bit quantization (NF4) and 8-bit AdamW optimizer to reduce memory footprint. The LoRA configuration included a rank of 16 and an alpha of 32 (scaling factor 2.0). The model was trained for 4 epochs with a learning rate of 2e-5, a batch size of 2 (accumulated to 16), and a maximum sequence length of 3,269 tokens.
   • Loss Function: Following best practices, the loss calculation was masked for user/system prompts, focusing optimization solely on the assistant's generated response tokens to prevent the model from learning to predict the prompt.

3. Evaluation Framework:

   • A memory-efficient framework was developed to evaluate nine LLMs (including proprietary models like Gemini 3 Flash and GPT-5 Mini, and open-weights models like Krikri 8B and Plutus 8B) across nine Greek QA datasets.
   • Metrics: Multiple-choice tasks were evaluated using accuracy (via rule-based parsing), while the open-ended CulturaQA task utilized BERTScore F1.
   • Statistical Analysis: Significance was assessed using McNemar's exact test for binary accuracy and Wilcoxon signed-rank tests for continuous BERTScore, supplemented by bootstrap resampling for 95% confidence intervals.

Key Contributions
The paper makes four primary contributions:

1. CulturaQA: A high-quality, human-curated Greek dataset designed specifically for LLM training and evaluation, covering diverse cultural and domain-specific topics.
2. Maistros 8B: A state-of-the-art open-weights Greek LLM developed via knowledge distillation and fine-tuning on CulturaQA, optimized for local deployment on commodity hardware.
3. Evaluation Framework: A memory-efficient, adaptable framework for evaluating LLMs on diverse Greek QA tasks using standard metrics.
4. Comprehensive Benchmarking: A rigorous evaluation of nine LLMs across nine human-curated datasets, providing a comparative analysis of open-weights versus proprietary models in the Greek context.

Results
Empirical evaluation across the nine datasets yielded the following findings:

• Proprietary vs. Open-Weights: Proprietary models (Gemini 3 Flash and GPT-5 Mini) consistently outperformed all open-weights models across most benchmarks.
• Maistros 8B Performance: Maistros 8B achieved state-of-the-art performance among open-weights models on most datasets, demonstrating meaningful improvements over its base model (Ministral 3 8B).
  • It achieved statistically significant improvements over the base model in 5 out of 9 datasets (e.g., Plutus QA, Greek ASEP MCQA, Greek MMLU).
  • On the domain-specific Plutus QA (Greek economy), Maistros 8B outperformed all other open-weights models and narrowed the gap with GPT-5 Mini to a 3.56% difference.
  • On the CulturaQA test set, Maistros 8B matched the performance of proprietary models closely while significantly outperforming other open-weights alternatives.
• Limitations: Maistros 8B did not show statistically significant improvements over the base model on Demos QA and Greek Truthful QA, and it remained below the performance of proprietary models on all benchmarks.

Significance and Claims
The authors claim that this work demonstrates the viability of using high-quality, LRM-generated, and human-curated synthetic data to train robust open-weights models for under-resourced languages. The study highlights that targeted fine-tuning on culturally grounded data can significantly enhance model performance in specific domains (such as finance) and general cultural understanding.

However, the paper maintains a modest stance regarding the gap between open and proprietary models. The authors acknowledge that while Maistros 8B represents a significant step forward for open-source Greek NLP, it does not yet match the capabilities of proprietary models like Gemini 3 Flash. The work serves as a foundation for future research, suggesting that further expansion of datasets to include long-form answers, diverse content, and regional dialects is necessary to fully capture the complexity of Modern Greek. The authors emphasize that their release of code, model, and data aims to support reproducibility and the development of linguistically accurate, culturally grounded language models for Greek.