🤖 machine learning

LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language

The paper introduces LLiMba, a 3B-parameter Sardinian language model adapted on a single consumer GPU via continued pretraining and fine-tuning, demonstrating that a high-rank rsLoRA configuration outperforms other adaptation methods in translation quality while highlighting critical limitations of standard metrics in capturing qualitative failures like script leakage and factual hallucinations.

Original authors: Luca Ballore

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luca Ballore

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, multilingual student (the AI model) who has read almost every book in the library of major languages like English, Spanish, and Italian. However, this student has never heard of Sardinian, a beautiful but endangered language spoken by about a million people on an island in Italy. When you ask this student to speak Sardinian, they just guess and accidentally speak Italian or Spanish instead.

The paper "LLiMba" is the story of how the author taught this student to speak Sardinian fluently using only a single, standard home computer graphics card (a 24 GB GPU), rather than a massive supercomputer.

Here is how they did it, broken down into simple steps:

1. The Two-Step Training Camp

To teach the student Sardinian, the author used a two-stage training process:

Stage 1: The "Immersive Reading" (Continued Pretraining)
First, the student was given a massive library of Sardinian text (about 11.5 million words) to read. This included news, Wikipedia articles, books, and poetry. To keep the student from forgetting how to speak Italian or Spanish, they also mixed in some related Romance language texts (like Italian and Spanish) as "replay" material.
- The Result: After this reading phase, the student could understand Sardinian much better. If you asked them to translate English to Sardinian, they went from being almost useless to being decent.
Stage 2: The "Conversation Practice" (Supervised Fine-Tuning)
Reading is good, but speaking is better. The student then practiced answering specific questions and following instructions in Sardinian. The author tested five different ways to teach this conversation skill to see which method worked best without needing too much computer memory.

2. The Five Teaching Methods (The "Adapter" Experiment)

The author didn't just retrain the whole student from scratch (which would be too heavy for the computer). Instead, they tried five different "training hats" (techniques called adapters) to tweak the student's brain:

Full Fine-Tuning: Rewriting the student's entire brain. (Heavy, but a good baseline).
LoRA (Rank 64): A small, lightweight hat.
rsLoRA (Rank 128): A slightly bigger, "stabilized" hat.
rsLoRA (Rank 256): The biggest, most powerful lightweight hat.
DoRA (Rank 256): A fancy hat that tries to keep the student's original personality intact while adding new skills.

3. The Surprising Results

The author found some counter-intuitive things:

Bigger isn't always better (for the "DoRA" hat): The "DoRA" hat was designed to be very careful and preserve the student's original knowledge. However, it actually performed the worst at telling the truth. It spoke fluently but made up wild, confident lies about history and culture.
The "rsLoRA" Hat Won: The biggest lightweight hat (rsLoRA with rank 256) was the champion. It produced the best translations and, crucially, told the most accurate facts.
The "Rank" Matters More Than the "Type": The study found that simply making the adapter "bigger" (increasing the rank) mattered more than choosing between the different types of hats. The biggest hat (rsLoRA 256) beat the smaller ones and even the full retraining method for translation tasks.

4. The "Hallucination" Trap

The paper highlights a tricky problem that standard tests (which just count how many words match) miss: Confident Lies.

The Fact-Checking Test: The author asked the models specific questions, like "Who was Gigi Riva?" (a famous Sardinian footballer).
- The rsLoRA model got the facts right.
- The LoRA model made up a fake biography (claiming he won the World Cup in Scotland, which is impossible).
- The DoRA model made up even wilder lies (claiming he played for teams he never played for and that a stadium was renamed after him when it wasn't).
The "Fake Words" Test: When asked to write a long, open-ended essay about Sardinian culture, the best model (rsLoRA) sometimes invented words that sounded like Sardinian but didn't actually exist. It's like a student who knows the rules of grammar so well they can invent a new word that sounds perfect but means nothing.

5. The Lesson on "Scores"

The paper warns us not to trust "perplexity" scores (a common math metric used to judge AI) too much, especially for languages that use different scripts.

The Analogy: Imagine grading a student on a test where the answers are written in a code. If the code is very predictable, the student might get a perfect score just by guessing the pattern, even if they don't understand the meaning. The paper shows that for some languages, the math score looks great, but the actual translation quality is terrible. You need real humans to check if the AI is actually telling the truth.

Summary

The paper proves that you can teach a small AI model to speak a rare language like Sardinian on a single home computer. The secret sauce was using a specific, larger "adapter" (rsLoRA) that balanced memory and creativity well. However, even the best model still has a tendency to confidently make things up, proving that we need human fact-checkers, not just math scores, to judge if an AI is truly helpful.

Technical Summary: LLiMba — Adapting a 3B Language Model to Sardinian

Problem Statement
Sardinian (sardu), a Romance language spoken by approximately one million people, suffers from a near-total absence in modern Natural Language Processing (NLP). Commercial services do not support it, and existing large language models (LLMs) fail to generate it reliably, often defaulting to Italian, Spanish, or other major Romance languages. The scarcity of Sardinian data in web-scale corpora prevents standard open models from learning the language, while proprietary models deem the user base too small for data acquisition. Furthermore, standard low-resource datasets (e.g., OSCAR, Leipzig Corpora) contain negligible usable Sardinian text. The challenge is to adapt a pre-trained multilingual model to a low-resource, closely related Romance language using minimal data and consumer-grade hardware, while avoiding catastrophic forgetting and ensuring factual grounding.

Methodology
The authors present LLiMba, a 3-billion-parameter Sardinian-ready model adapted from Qwen2.5-3B-Instruct. The adaptation pipeline is executed on a single NVIDIA RTX 4090 (24 GB VRAM) using a two-stage process:

Continued Pretraining (CPT):
- Data: A corpus of ~11.5 million Sardinian tokens (spanning LSC, Logudorese, and Campidanese variants) augmented with ~2.4 million tokens of related Romance text (Italian, Spanish, Portuguese, Catalan) to prevent catastrophic forgetting and register blurring.
- Technique: Full fine-tuning (all parameters updated) in bfloat16. The authors chose full fine-tuning over parameter-efficient adapters for this stage because the domain shift (learning a new language) is large, and prior work suggests LoRA underperforms full fine-tuning when the target distribution is distant from pretraining.
- Hardware Constraints: To fit the 3B model on 24 GB VRAM, the authors utilized paged 8-bit AdamW optimizers, gradient checkpointing, and an effective batch size of 16 via gradient accumulation. Sequence packing was disabled to prevent attention leakage across heterogeneous document types.
Supervised Fine-Tuning (SFT):
- Data: ~14,404 instruction pairs derived from machine-translated Capybara data, parallel translation pairs, native-curated synthesized instructions, and song-related queries.
- Configuration Comparison: Five SFT configurations were compared under matched conditions (same data, hardware, hyperparameters):
  - Full Fine-Tuning.
  - LoRA (Rank 64, $\alpha=128$ ).
  - rsLoRA (Rank 128, $\alpha=128$ , using $\alpha/\sqrt{r}$ scaling).
  - rsLoRA (Rank 256, $\alpha=256$ , using $\alpha/\sqrt{r}$ scaling).
  - DoRA (Rank 256, $\alpha=256$ , decomposing weights into magnitude and direction).

Key Results

Translation Performance: Continued pretraining provided the majority of the performance gain, improving BLEU scores by 4–7x over the base model. SFT provided incremental improvements.
- Best Performer: rsLoRA at Rank 256 achieved the highest scores for translation into Sardinian (e.g., 28.47 BLEU for English-to-Sardinian, compared to 17.26 after CPT alone and 21.04 for full fine-tuning).
- Rank Ablation: Performance on translation metrics scaled smoothly with rank: rsLoRA r256 > rsLoRA r128 > LoRA r64.
- DoRA Underperformance: Despite having the same rank (256) as the best rsLoRA configuration, DoRA underperformed LoRA r64 on four of six translation directions and failed to match the gains of full fine-tuning.
Qualitative Findings & Failure Modes:
- Factual Grounding: Translation metrics (BLEU/chrF) did not correlate with factual accuracy. rsLoRA r256 produced verifiable, accurate biographical and cultural facts. In contrast, LoRA r64 produced confident but fabricated details, and DoRA r256 produced the most aggressive fabrications, contradicting its design premise of preserving base model structure.
- Lexical Fidelity: While rsLoRA r256 was factually superior, it exhibited "morphological hallucination" on long, open-ended prompts, generating phonotactically plausible but unattested Sardinian words. Full fine-tuning remained closest to attested vocabulary.
- Script Leakage: The rsLoRA r128 configuration introduced a unique failure mode: leakage of non-Latin script tokens (Chinese/English) into Sardinian sentences, a phenomenon not observed in other variants.
Loss Metrics vs. Reality: Training and evaluation losses were similar across methods and did not predict the qualitative differences in factuality or script integrity. The paper notes that perplexity metrics can be artificially deflated by byte-fallback tokenization in non-Latin scripts (a known issue in Tibetan adaptation), though Sardinian's Latin script avoids this specific artifact.

Significance and Contributions
The paper establishes that adapting a 3B parameter model to a low-resource Romance language is feasible on a single consumer GPU using a two-stage pipeline of full CPT followed by parameter-efficient SFT.

Key contributions include:

Empirical Evidence on Adapter Capacity: The study demonstrates that for adapting a Romance-pretrained base to a low-resource Romance target, adapter rank matters more than the specific adapter variant. Scaling rsLoRA to Rank 256 yielded significant gains, while DoRA at the same rank failed to deliver comparable results, suggesting that the specific parameterization (unconstrained low-rank updates vs. magnitude-direction decomposition) interacts critically with the task.
Limitations of Standard Metrics: The work highlights that translation metrics (BLEU/chrF) and loss curves can order configurations smoothly while masking categorical differences in qualitative behavior, such as factual hallucination and script leakage.
Resource Efficiency: It provides a reproducible pipeline for low-resource language adaptation that does not require massive compute clusters, relying instead on careful data curation and hardware-aware optimization (8-bit optimizers, gradient checkpointing).

The authors conclude that while rsLoRA r256 is the optimal configuration for translation and factual grounding in this specific setting, the divergence between quantitative metrics and qualitative reality necessitates evaluation strategies that include native speaker probes and factuality checks, particularly when loss-based metrics may be misleading or insufficient.