💬 NLP

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

This paper introduces LuxIT, a high-quality monolingual instruction-tuning dataset for Luxembourgish synthesized from native texts and validated via LLM-as-a-judge, demonstrating that fine-tuning smaller models on this data significantly improves their performance on language proficiency exams and various downstream NLP tasks.

Original authors: Julian Valline, Cedric Lothritz, Siwen Guo, Jordi Cabot

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Julian Valline, Cedric Lothritz, Siwen Guo, Jordi Cabot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, multilingual robot chef. This chef can cook amazing meals in English, French, and German because it has read millions of cookbooks in those languages. But if you ask it to cook a traditional Luxembourgish stew, it stammers. It might mix in German spices or French herbs, but it can't quite get the authentic flavor right.

Why? Because nobody has ever written a "Luxembourgish Cookbook" for it to study. The robot is starving for specific instructions in that language.

This paper, LuxIT, is the team that finally wrote that cookbook. Here is the story of how they did it, explained simply.

1. The Problem: The "Language Desert"

Most AI models are like students who only speak English. They are great at answering questions in English but struggle with "low-resource" languages like Luxembourgish (spoken by about 600,000 people).

Usually, to teach an AI a new language, you need thousands of human-written examples of "Question + Answer." But for Luxembourgish, those examples are rare. It's like trying to teach someone to play the violin using only one sheet of music.

2. The Solution: The "Synthetic Chef"

Instead of waiting for humans to write millions of examples (which would take forever), the authors used a clever trick: They used a super-smart AI to write the examples for another AI.

The Source Material: They gathered a massive library of real Luxembourgish text from Wikipedia and news articles (RTL). Think of this as a giant pile of raw ingredients.
The Generator: They used a powerful AI model called DeepSeek-R1 (which is already quite good at Luxembourgish) to act as a "Synthetic Chef."
The Recipe: They told the Chef: "Read this news article. Now, invent three different questions a human might ask about it, and write the perfect answers in fluent Luxembourgish."

The result? The Chef cooked up 227,507 new "Instruction-Answer" pairs. It's like a factory that can print out millions of practice quizzes in seconds.

3. The Quality Control: The "Strict Food Critic"

You can't just trust a robot to write perfect recipes; it might hallucinate or use bad grammar. So, the team added a "Food Critic" step.

The Critic: They used another AI (GPT-5-mini) to taste-test every single generated pair.
The Grading: The critic gave scores on four things:
1. Language: Does it sound like a real Luxembourger wrote it?
2. Facts: Is the answer actually true?
3. Following Orders: Did it answer the specific question asked?
4. Helpfulness: Is it actually useful?
The Filter: Any "dish" that wasn't perfect was thrown in the trash. They kept only the high-quality ones.

Note: The team also had real humans taste-test a few samples. Interestingly, the human critics were stricter than the AI critic, but they agreed that the final batch of food was generally delicious.

4. The Test: Putting the Robots to School

Now that they had their new "Luxembourgish Cookbook" (the LuxIT dataset), they tested it on 14 different AI models (ranging from small to medium-sized).

They treated these models like students taking a final exam. The exams covered:

Language Proficiency: Grammar, vocabulary, and reading comprehension (like a school test).
Real-World Tasks: Sentiment analysis (is this text happy or sad?), understanding instructions, and logic puzzles.

5. The Results: A Major Upgrade

The results were like watching a student go from failing a class to getting an A.

The Big Win: For 12 out of 14 models, their Luxembourgish skills improved significantly. On average, their test scores went up by 5.37 percentage points.
The Star Student: One small model (Ministral-3-3B) jumped from a 25% score to a 46% score. That's a massive leap!
The Surprising Twist: Interestingly, getting better at "school exams" (grammar tests) didn't always mean getting better at "real-world tasks" (like analyzing sentiment). It's like a student who can pass a grammar test perfectly but still struggles to write a funny joke. This shows that language is complex; being fluent in one area doesn't automatically fix everything.

6. Why This Matters

This paper proves a powerful idea: You don't need millions of human writers to teach an AI a rare language.

If you have some text in that language (like news and Wikipedia), you can use a smart AI to generate the rest of the training data. It's a "force multiplier" for low-resource languages.

In a nutshell:
The team built a machine that reads Luxembourgish news, invents practice questions, grades its own work, and then uses that practice material to teach other robots how to speak Luxembourgish. The result? A whole generation of AI models that can finally understand and speak the language of Luxembourg much better than before.

1. Problem Statement

Large Language Models (LLMs) have achieved remarkable proficiency in high-resource languages (primarily English) through instruction tuning (IT). However, low-resource languages like Luxembourgish suffer from a scarcity of high-quality, open-source instruction tuning datasets. Existing datasets are predominantly English or rely on machine translation, which often fails to capture the nuances of low-resource languages. Furthermore, creating resources for Luxembourgish is challenging due to the lack of parallel corpora (e.g., Luxembourgish-English pairs) required for cross-lingual approaches. The authors aim to address this gap by creating a high-quality, monolingual instruction tuning dataset derived exclusively from native Luxembourgish text, without relying on machine translation or parallel data.

2. Methodology

The authors propose a five-step pipeline to generate the LuxIT dataset, utilizing a "synthetic data generation" approach powered by a strong LLM.

A. Data Collection & Preprocessing

Seed Data: The pipeline starts with two monolingual sources:
1. A dump of the Luxembourgish Wikipedia (approx. 80k articles).
2. A collection of RTL Luxembourg news articles and comments (up to May 2024).
Filtering: To ensure quality, the authors exclude user comments (due to brevity and noise) and apply heuristic filters to the news and Wikipedia data. This includes removing articles shorter than 750 characters, excluding list/stub pages, and filtering out non-Luxembourgish content.
Final Seed Set: The refined corpus consists of ~16.5k Wikipedia articles and ~100k RTL news articles.

B. Synthetic Instruction Generation

Generator Model: The authors selected DeepSeek-R1-0528, identified as the top-performing model for Luxembourgish comprehension and generation.
Prompt Engineering: The model was prompted to generate 3 instruction-answer pairs for each seed text. The prompts enforced strict constraints:
- All content must be in fluent, natural Luxembourgish.
- Instructions must be self-contained (embedding necessary context).
- Diversity in task types: Summarization, Question Answering, Information Extraction, and Explanation.
- For summarization, the original text must be included in the instruction.
Output: This step initially produced 245,624 synthetic instruction-answer pairs.

C. Quality Assurance (LLM-as-a-Judge)

Automated Evaluation: The generated pairs were evaluated using GPT-5-mini as a judge.
Scoring Criteria: Each pair was scored (1–3 scale) on four dimensions:
1. Linguistic Quality: Grammar, spelling, and naturalness.
2. Factual Accuracy: Consistency with the source text.
3. Instruction Adherence: Following the prompt constraints.
4. Helpfulness & Relevance: Usefulness of the instruction.
Filtering: Only pairs scoring 2 or higher on all four metrics were retained.
Human Validation: A subset of 100 samples was manually evaluated by 3 native speakers. While human retention was lower (69%) than the LLM judge (92.6%), the human evaluation confirmed high factual accuracy (94.2% rated "Excellent") despite low inter-annotator agreement (Fleiss' $\kappa$ = -0.019 for factual accuracy), highlighting the subjectivity of evaluating synthetic Luxembourgish.

D. Final Dataset

LuxIT: The final dataset contains 227,507 high-quality instruction-answer pairs.
Retention Rate: 92.62% of the generated samples were retained.

3. Experimental Setup

To validate the utility of LuxIT, the authors fine-tuned 14 different LLMs (ranging from 0.5B to 12B parameters) on the dataset.

Models Tested: Includes Llama-3.1/3.2, Gemma-3, GLM-4, Qwen2.5, Mistral, Phi-4, Apertus, EuroLLM, and Olmo-3.
Training: Fine-tuning was performed using LoRA (Low-Rank Adaptation) with the Unsloth framework.
Evaluation Benchmarks:
1. Language Proficiency Exams: Standardized tests from the Institut National des Langues Luxembourg (INLL) covering CEFR levels A1–C2 (Vocabulary, Grammar, Reading, Conversation).
2. Downstream NLP Tasks: Five standard tasks (Intent Classification, RTE, Sentiment Analysis, SST-2, WNLI) adapted for Luxembourgish.

4. Key Results

A. Language Proficiency (RQ1 & RQ2)

Overall Improvement: Fine-tuning on LuxIT led to a mean accuracy increase of +5.37 percentage points (pp) across all 14 models on language exams.
Model Performance:
- 12 out of 14 models improved.
- Best Performer: Ministral-3-3B-Instruct saw a massive gain of +21.0 pp (from 25.8% to 46.8%).
- Top Absolute Performer: GLM-4-9B-0414 achieved the highest final macro-average accuracy (60.6%).
- Degradation: Two small models (Gemma-3-1B and Qwen2.5-0.5B) suffered significant performance drops (-22.0 pp and -13.6 pp, respectively), suggesting potential catastrophic forgetting or architectural incompatibility at very small scales.
Skill Categories: Improvements were most consistent in Vocabulary and Conversational Comprehension. Gains were less pronounced in Grammar, and the C2 (Native) level saw the most inconsistent results, with 6 models declining, suggesting the synthetic data may be skewed toward intermediate (B1/B2) proficiency.

B. Downstream NLP Tasks (RQ3)

Mixed Results: On the 5 downstream NLP tasks, 9 out of 14 models improved in macro-averaged F1 scores.
Lack of Correlation: There was no systematic correlation between gains in language exams and gains in NLP tasks.
- Example: Ministral-3-3B had the largest exam gain (+21.0 pp) but near-zero NLP task change (-0.003).
- Example: Phi-4 showed massive NLP gains (+0.325) but moderate exam gains.
Interpretation: The authors suggest that LuxIT improves formal comprehension and fluency (aligned with the standardized exam style) but does not necessarily transfer to structured label prediction on noisy, real-world text (NLP tasks).

5. Key Contributions

LuxIT Dataset: The first instruction tuning dataset for Luxembourgish created exclusively from monolingual seed data, bypassing the need for parallel corpora.
Synthetic Data Pipeline: A validated pipeline using DeepSeek-R1-0528 and LLM-as-a-judge to generate high-quality synthetic data for low-resource languages.
Comprehensive Evaluation: A rigorous benchmarking of 14 models across standardized language exams and diverse NLP tasks, revealing the nuanced impact of instruction tuning on low-resource capabilities.
Empirical Insights: Demonstration that synthetic monolingual data can significantly boost LLM performance in low-resource settings, though the benefits vary by model architecture and task type.

6. Significance and Limitations

Significance: This work proves that high-quality instruction tuning datasets for low-resource languages can be synthesized without expensive human annotation or parallel data. It provides a blueprint for other low-resource language communities.
Limitations:
- Domain Specificity: The data is derived from Wikipedia and news, potentially lacking conversational, dialectal, or creative styles.
- Evaluation Bias: The evaluation relies on multiple-choice exams, which may not fully capture open-ended conversational abilities.
- Judge Bias: The automated judge (GPT-5-mini) showed a leniency bias compared to human evaluators.
- Small Model Instability: The degradation observed in sub-1.5B models highlights the risk of catastrophic forgetting when fine-tuning very small models on synthetic data.

In conclusion, LuxIT represents a significant step forward for Luxembourgish NLP, validating that monolingual synthetic data generation is a feasible and effective strategy for enhancing LLM capabilities in low-resource linguistic environments.