💬 NLP

DIETA: A Decoder-only transformer-based model for Italian-English machine TrAnslation

This paper introduces DIETA, a 0.5-billion-parameter decoder-only Transformer model trained on a curated 207-million-sentence Italian-English corpus and back-translated data, which achieves competitive performance on benchmarks and is accompanied by the public release of its training scripts, models, data, and a new evaluation set.

Original authors: Pranav Kasela, Marco Braga, Alessandro Ghiotto, Andrea Pilzer, Marco Viviani, Alessandro Raganato

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pranav Kasela, Marco Braga, Alessandro Ghiotto, Andrea Pilzer, Marco Viviani, Alessandro Raganato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a translator that speaks only two languages: Italian and English. Most big tech companies build "universal translators" that try to speak 100 languages at once. These are like massive, heavy backpacks filled with every dictionary in the world. They are powerful, but they are also huge, expensive to run, and sometimes get confused because they are trying to remember too many things at once.

The authors of this paper decided to build something different: a specialized, lightweight translator called DIETA.

Here is the story of how they built it, using simple analogies:

1. The Goal: A Specialized Athlete, Not a Marathon Runner

Instead of training a giant model to speak every language (like a marathon runner who tries to run every distance), the team trained a small, 0.5-billion-parameter model to be the absolute best at just Italian and English.

The Metaphor: Think of DIETA as a sprinter. It doesn't need to carry the weight of 100 languages; it just needs to be incredibly fast and accurate at running the Italian-English track.

2. The Training Data: The "Gym"

To get good at anything, you need practice. The team didn't just use a few textbooks; they built a massive gym with 768 million practice sentences.

The Real Stuff: They gathered about 207 million real sentence pairs from places like parliamentary records, legal documents, news, movies (subtitles), and books.
The "Ghost" Practice (Back-Translation): Since they needed even more practice, they used a clever trick called "back-translation." Imagine taking an Italian news article, translating it to English with a computer, and then having the computer translate that English back into Italian. This creates a new "synthetic" practice pair. They did this millions of times to create a massive library of 352 million extra sentences.
The Result: The model practiced on a mix of real human writing and this massive amount of computer-generated practice data.

3. The New Test: The "Fresh News" Exam

Usually, translation models are tested on old, static data. The authors realized this doesn't tell you if a model can handle today's news.

The Innovation: They created a new test set called WikiNews-25, based on articles from 2025.
The Catch: They didn't just use any sentences. They took translations made by Google, found the ones that were wrong, and had humans fix them. This created a "gold standard" exam specifically designed to test how well a model handles current, real-world news.

4. The Results: Punching Above Its Weight

They put DIETA in a race against 32 other translators, ranging from tiny models to massive ones with 9 billion parameters (like the heavy backpacks mentioned earlier).

The Performance: Even though DIETA is tiny (only 0.5 billion parameters), it consistently finished in the second-best group (the second quartile).
The Comparison: It beat almost every other model that was smaller than 3 billion parameters. In fact, on four out of five different tests, it performed better than almost all the other small models.
The Trade-off: It wasn't quite as perfect as the giant 9-billion-parameter models (the "super-athletes"), but it was incredibly close. The main area where the giants still won was in "reference-free" scoring (guessing quality without a perfect answer key), but for everything else, DIETA held its own.

5. Why This Matters

The paper claims that you don't always need a massive, expensive supercomputer to get great translation results.

The Takeaway: If you focus your training data specifically on two languages and use a lot of smart, synthetic practice data, a small, lightweight model can do a job that usually requires a much bigger, heavier one.
Accessibility: Because DIETA is so small, it can run on a single consumer graphics card (like a high-end gaming PC), whereas the giant models require massive data centers.

Summary

The authors built DIETA, a small, specialized translator for Italian and English. They trained it on a massive mix of real documents and computer-generated practice sentences. They tested it on a brand-new, 2025 news dataset. The result? A tiny model that performs nearly as well as the giants, proving that specialized training and smart data can beat raw size.

They have made the model, the training data, and the new test set available to everyone so others can learn from it and build even better tools.

Technical Summary: DIETA – A Decoder-only Transformer for Italian–English Machine Translation

Problem Statement

Despite the advancements of Transformer-based Large Language Models (LLMs) in Natural Language Processing, high-quality neural machine translation (MT) for specific language pairs, particularly Italian–English, remains an open challenge. General-purpose multilingual systems often prioritize broad language coverage over specialized translation quality, leaving room for improvement in targeted pairs. Furthermore, massive multilingual models incur high computational costs and may suffer from representational competition across hundreds of languages, potentially underperforming on specific directions like English–Italian compared to smaller, specialized models.

Methodology

Data Collection and Curation

The authors constructed a massive training corpus comprising approximately 768 million source–target examples, derived from three primary sources:

Parallel Corpora: Approximately 207.8 million high-quality sentence pairs were curated from public resources in the OPUS repository, including Europarl, DGT-TM, ParaCrawl, OpenSubtitles, WikiMatrix, and legal texts.
- Cleaning: Duplicates were removed using OpusTools/OpusFilter. Remaining pairs underwent quality filtering using the Phi-4 LLM with a binary prompt to verify translation equivalence, discarding non-matching pairs.
- Formatting: Pairs were duplicated with explicit direction tags (e.g., ENG: ... IT: ...) to support bidirectional training.
Synthetic Data (Back-Translation): To augment the corpus, the authors generated 352 million synthetic sentence pairs via back-translation:
- NewsCrawl: Italian (2008–2018) and English (2023) articles were translated using OPUS-MT-TC-BIG; newer segments (2019–2024) used NLLB-200-3.3B.
- FineWeb: 108.5 million Italian and 100 million English sentences were translated using the multilingual GemmaX2-28-9B-v0.1 model.
- All synthetic translations were generated using the CTranslate2 toolkit in greedy decoding mode.

Model Architecture

DIETA is a decoder-only Transformer model with 0.5 billion parameters, specifically designed for bidirectional Italian–English translation.

Layers: 6 identical layers with a post-norm configuration.
Dimensions: Hidden dimension of 2,048 and 32 attention heads.
Feed-Forward: Uses squared-ReLU activation with a 4x expansion factor.
Embeddings: Rotary embeddings for token positions.
Optimizations: Incorporates residual attention accumulation and query-key normalization.
Tokenizer: Utilizes a 51,200-entry SentencePiece vocabulary from the Minerva family, trained on a balanced corpus of high-quality Italian and English to optimize sub-word segmentation for these specific languages.

Training Strategy

Models were trained using the x-Transformers framework with the Lion optimizer (learning rate $2 \times 10^{-4}$ ) over a single epoch with a linear decay schedule following a 10% warm-up. Five model variants were released to analyze the impact of data composition:

DIETA: Trained from scratch on the filtered parallel corpus (415.7M pairs).
DIETA+BT: Trained on parallel corpus + NewsCrawl back-translations.
DIETA+cont: Continued training of DIETA for a second epoch on the mixed parallel + NewsCrawl data.
DIETA+nosynth: Continued training of DIETA for a second epoch on parallel data only.
DIETA+allsynth: Continued training of DIETA+cont for a third epoch on the full corpus (Parallel + NewsCrawl + FineWeb), totaling 768M pairs.

Evaluation

The models were evaluated against 15+ baselines (including NLLB-200, OPUS-MT, Minerva, LLaMAntino, Gemma-2, and Tower) across five benchmarks:

Standard Benchmarks: NTREX-128, Tatoeba, WMT-24pp, and FLORES-200.
New Benchmark: WikiNews-25, a 450-sentence test set based on 2025 WikiNews articles, featuring human post-edited translations to assess performance on contemporary text.
Metrics: A comprehensive suite including surface overlap (sacreBLEU, chrF), neural reference-based (BLEURT, MetricX-24, COMET), and reference-free quality estimation (QE MetricX, COMETKiwi).

Key Results

Performance vs. Size: DIETA consistently ranks in the second quartile of a 32-system leaderboard. It outperforms most sub-3B models on four out of five test suites and matches or surpasses models 2–3 times larger (e.g., 1–3B parameter models) in many metrics.
Specific Benchmarks:
- NTREX-128: DIETA+cont achieves competitive BLEU (36/43) and COMET (0.85/0.87) scores, rivaling NLLB-1.3B and OPUS-MT-big.
- Tatoeba: DIETA+cont scores 58/70 BLEU, surpassing all competitors below 3B parameters and trailing only the largest models (Gemma-9B, PhiMaestra-3) by a small margin.
- WMT-24pp & WikiNews-25: The model delivers mid-table performance competitive with 7B systems, particularly on recent news data (WikiNews-25), where it equals or surpasses systems below 7B parameters.
- FLORES-200: DIETA+allsynth virtually matches NLLB-3.3B in COMET scores despite having one-sixth the parameters.
Data Impact: Synthetic back-translation data provided clear gains. DIETA+BT added +1 to +3 BLEU over the parallel-only baseline, and continued training (DIETA+cont, DIETA+allsynth) added further improvements without increasing model size.
Directionality: The IT→EN direction generally performed better (2–12 BLEU higher) due to richer target-side data, though this gap narrowed with the introduction of synthetic Italian data. WikiNews-25 was an outlier where EN→IT was slightly easier.

Significance and Contributions

The paper claims that data scale and task-specific training can compensate for a significant reduction in model parameters. The primary contributions are:

DIETA Model: A specialized, open-source, 0.5B-parameter decoder-only Transformer that delivers robust, high-quality Italian–English translation, fitting on a single consumer GPU while remaining competitive with much larger LLMs.
Curated Corpus: The release of a large-scale, carefully curated parallel corpus (207M pairs) and a synthetic corpus (352M pairs) derived from diverse sources.
WikiNews-25: A new evaluation set based on 2025 news articles to facilitate benchmarking on contemporary, human-corrected content.
Efficiency: Demonstrating that a small, specialized model can achieve performance comparable to larger general-purpose models, suggesting that future gains may lie in reference-free quality estimation (QE) tuning rather than simply increasing model size.

The authors conclude that while a performance gap remains in reference-free QE metrics compared to 9B state-of-the-art models, the DIETA family offers a highly efficient solution for specialized translation tasks, with future work planned to address the QE gap via fine-tuning and to enable edge deployment through quantization.