💻 computer science

Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfer

This paper introduces Luth, a family of French-specialized Small Language Models that achieve state-of-the-art performance on French benchmarks while retaining English capabilities through targeted post-training and strategic model merging, effectively addressing the performance gap in non-English SLMs.

Original authors: Maxence Lasbordes, Sinoué Gad

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maxence Lasbordes, Sinoué Gad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence (AI) as a massive library. For a long time, almost all the books in this library have been written in English. If you ask the librarian (the AI) a question in French, they might stumble, give a vague answer, or sound like they are reading a dictionary rather than having a conversation. This is especially true for the "smaller" librarians (Small Language Models or SLMs), who are designed to be fast and efficient but often lack deep knowledge of languages other than English.

The paper you shared introduces a new team of librarians called Luth. Their goal was simple: take these efficient, small AI models and teach them to speak French fluently without making them forget how to speak English.

Here is how they did it, using some everyday analogies:

1. The Problem: The "English-Only" Bias

Most AI models are trained on a diet of mostly English data. It's like teaching a student only in English; they might be great at English literature but struggle with French history or math problems written in French. The authors noticed that even the best "multilingual" models were still significantly weaker in French compared to English.

2. The Solution: A Specialized "French Boot Camp" (Luth-SFT)

To fix this, the authors didn't just throw more random French text at the AI. Instead, they built a specialized training camp called Luth-SFT.

The Curriculum: They gathered 570,000 high-quality "instruction and answer" pairs. Think of this as a massive workbook of French questions and perfect answers.
The Translation Trick: They took excellent English textbooks (datasets) and didn't just translate the answers word-for-word. Instead, they translated the questions into French and then asked the AI to write new answers from scratch. This ensured the French sounded natural and human, not robotic.
The "Scholar" Section: A special part of this workbook focused on hard academic subjects (like Math, Physics, and Engineering) using real exam papers from French high schools and universities. This gave the AI a serious "brain boost" in reasoning and logic.

3. The Training: Full Fine-Tuning

They took existing small AI models (the "base" models) and ran them through this French boot camp. This is like taking a generalist athlete and putting them through a rigorous, specialized training regimen to become a French-speaking expert.

The Catch: When you train a model intensely on one language, it sometimes starts to forget its other skills. In this case, the models got great at French but started to get slightly worse at English.

4. The Magic Trick: "Model Merging" (The Smoothie Effect)

This is where the paper gets clever. To fix the "forgetting English" problem without losing the new French skills, they used a technique called Model Merging.

Imagine you have two smoothies:

Smoothie A: The original model (Great at English, okay at French).
Smoothie B: The newly trained model (Amazing at French, slightly weaker at English).

Instead of choosing one or the other, they "blended" them together. They mixed the "brain" of the original model with the "brain" of the trained model.

The Result: The new blended model (Luth) kept the French superpowers and regained (or even improved) its English skills. It was like getting the best of both worlds in a single cup.

5. The Results: The New Champions

The authors tested these new Luth models against other small AI models on six different challenges (like math problems, following complex instructions, and general knowledge).

In French: The Luth models crushed the competition. They outperformed every other open-source model of the same size, improving scores by up to 11% on average.
In English: Surprisingly, they didn't just stay the same; they actually got better at English too. The authors call this "cross-lingual transfer," suggesting that learning French deeply actually helped the model understand English concepts better as well.

Summary

The paper claims that by creating a high-quality French dataset and using a "blending" technique to merge models, they created a family of small, efficient AI models that are now the best at French available, without sacrificing their English abilities. They proved you don't need a giant, expensive supercomputer to make a great French AI; you just need the right data and the right mixing method.

Technical Summary: Luth – Efficient French Specialization for Small Language Models

Problem Statement

The current landscape of Large Language Models (LLMs) remains predominantly English-centric, creating a significant performance gap for other major languages, particularly French. While multilingual models exist, they often underperform in French compared to English due to data imbalances. Furthermore, research into efficient adaptation methods for French, specifically within the domain of Small Language Models (SLMs), is limited. Existing French-specific efforts often rely on resource-intensive large-scale training or fail to match the performance of comparable multilingual baselines, leaving a need for compact, French-specialized models that can be efficiently adapted without sacrificing multilingual capabilities.

Methodology

The authors propose Luth, a family of French-specialized SLMs (ranging from 350M to 1.7B parameters) developed through a targeted post-training and model merging pipeline. The methodology consists of three primary stages:

1. Dataset Construction (Luth-SFT)

To address the scarcity of high-quality open-source French post-training data, the authors constructed the Luth-SFT dataset, comprising 570,000 samples (338 million tokens) of French instruction-response pairs. The data construction involved:

Gathering: Filtering French samples from existing multilingual datasets (AYA, Smoltalk2, CroissantLLM).
Translation and Generation: Selecting high-quality English instruction datasets (Tülu 3, OpenHermes). English prompts were translated into French using strong multilingual models (GPT-4o, Qwen3 32B). Crucially, new French responses were generated from scratch based on these translated prompts rather than translating original English answers, ensuring native linguistic quality.
Filtering: A two-stage pipeline enforced linguistic validation (grammar, coherence, no code-switching) and content filtering (removing code, tool-calling, and factual errors).
Scholar Subset: A specialized subset of 30,300 samples was created from French Baccalauréat and Classes Préparatoires (CPGE) examination materials (1980–2025). This non-synthetic data, covering subjects like Mathematics (67.23%), Physics-Chemistry, and Computer Science, was processed via LLM-assisted parsing to extract questions and detailed, expert-validated solutions.

2. Model Training

The authors evaluated several base multilingual models in the sub-2B parameter range, selecting Qwen3 (0.6B, 1.7B) and LFM2 (350M, 700M, 1.2B) based on their strong initial performance in French and English.

Full Fine-Tuning: Instead of parameter-efficient methods like LoRA, the selected models underwent full fine-tuning on the Luth-SFT dataset using the Axolotl framework.
Configuration: Training utilized a single NVIDIA H100 GPU with FlashAttention and sequence packing (max length 16,384). Hyperparameters included a learning rate of $2 \times 10^{-5}$ and a cosine scheduler with a 10% warm-up.

3. Model Merging

To mitigate the "catastrophic forgetting" of English capabilities often observed when fine-tuning exclusively on French data, the authors employed model merging.

Process: The French-fine-tuned models were merged with their original base models using MergeKit.
Techniques: The authors experimented with Linear Interpolation (LERP) and Spherical Linear Interpolation (SLERP).
Outcome: Surprisingly, simple interpolation methods yielded the most stable results. The merging coefficients (ranging from 0.3 to 0.7) were empirically selected to balance French specialization with English retention, often resulting in performance gains in both languages.

Key Contributions

Luth-SFT Dataset: A curated, high-quality French instruction-response dataset (570k samples) specifically designed to improve general knowledge, instruction following, and mathematical reasoning.
Luth Model Family: Five SLMs (350M to 1.7B parameters) that achieve state-of-the-art (SOTA) performance on French benchmarks within their size categories.
Efficient Adaptation Methodology: A reproducible pipeline demonstrating that targeted post-training combined with strategic model merging can significantly enhance language-specific capabilities while preserving or even improving cross-lingual performance.

Results

The Luth models were evaluated on six benchmarks (IFEval, GPQA-Diamond, MMLU, Math500, Arc-Challenge, HellaSwag) in both French and English.

French Performance: Luth models substantially outperformed all open-source counterparts of comparable size. They achieved an absolute average improvement of up to +11.26% across six French benchmarks compared to their base models. For instance, Luth-1.7B-Instruct scored 58.53 on IFEval (French) and 62.60 on Math500 (French), significantly outperforming the base Qwen3-1.7B.
English Performance: Contrary to the typical degradation seen in language-specific fine-tuning, the Luth models maintained stable or improved performance in English. The authors attribute this to cross-lingual transfer, where improvements in French reasoning and instruction following positively impacted English capabilities. Improvements in English ranged from +0.76% to +3.20% across the selected benchmarks.
Merging Efficacy: The experiments confirmed that merging the fine-tuned model with the base model not only recovered lost English performance but often enhanced overall results in both languages compared to the fine-tuned model alone.

Significance and Claims

The paper positions Luth as a new state-of-the-art baseline for French Small Language Models. The authors claim that their work demonstrates:

Efficiency: High-quality French capabilities can be achieved through careful post-training on curated data and model merging, avoiding the need for costly continual pre-training or massive-scale resources.
Robustness: Specialization in one language does not necessitate the sacrifice of others; in fact, strategic merging can lead to mutual improvements across languages.
Extensibility: The methodology is presented as a practical, reproducible recipe that can be extended to other languages and domains.

The authors acknowledge limitations, noting that their evaluation is restricted to a specific set of benchmarks, does not cover tool use or code generation, and is currently limited to models under 2 billion parameters. They suggest that verifying the scalability of these gains to larger architectures remains a direction for future work.