LuxMT Technical Report

1. Problem Statement

The primary challenge addressed in this paper is the lack of high-quality Machine Translation (MT) systems for Luxembourgish (LB), a low-resource language. Specifically, there is a need for:

• Specialized Models: Existing Large Language Models (LLMs) are not optimized for translating LB into major European languages like French (FR) and English (EN).
• Reliable Benchmarks: Standard benchmarks (e.g., FLORES-200) risk data contamination because base models are trained on web-scraped data that may inadvertently include these test sets.
• Evaluation Metrics: There are no metrics specifically trained for Luxembourgish, limiting the ability to perform reference-less quality estimation (QE).

2. Methodology

A. Benchmark Construction

To avoid data contamination, the authors constructed a custom benchmark using Luci, a quadrilingual tourist magazine for Luxembourg (editions 1–63).

• Data Processing: Text was segmented using spaCy and aligned using LUXEMBEDDER (Luxembourgish sentence embeddings).
• Filtering: Pairs were filtered for length (>5 words), duplicates, and quotation mark standardization.
• Selection: The top 500 segment pairs per language pair (LB-FR, LB-EN, LB-DE) were selected based on cosine similarity and manually verified to ensure context preservation.
• Note: The benchmark is currently not publicly available due to copyright restrictions.

B. Model Selection

The authors evaluated several popular local LLMs (GEMMA 3, AYA EXPANSE, COMMAND R, LLAMA 3.1/4, MISTRAL S, PHI 4) on the Luci benchmark.

• Result: GEMMA 3 (27B) emerged as the best-performing base model across all metrics (LUXEMBEDDER, BERTSCORE, BLEURT-20, XCOMETXL).

C. Data Curation & Fine-Tuning

• Training Sources:
  1. LUXALIGN: A parallel corpus of RTL news articles (LB-FR, LB-EN).
  2. Parliamentary Transcripts: Luxembourgish transcripts from the Chambre des Députés, augmented into FR and EN using Google Translate.
• Filtering Strategy: The authors used LUXEMBEDDER to filter low-equivalence segment pairs.
  • For LUXALIGN, a strict threshold of 0.99 was applied, reducing the corpus by ~85% but yielding the best results (likely because news articles are not 1-to-1 translations).
  • For augmented parliamentary data, a threshold of 0.98 was used to preserve more data, as these were 1-to-1 translations.
• Fine-Tuning Configuration:
  • Model: GEMMA 3.
  • Duration: 1 epoch (tested 1–3 epochs; 1 was optimal).
  • Learning Rate: 2e-5.
  • Prompt: "Translate from Luxembourgish to [Target Language]: [source segment]".
  • Total Data: ~54.5k segments (32k for LB→FR, 22.5k for LB→EN).

D. Evaluation Metrics

The study employed an ensemble of metrics to ensure robustness:

• Neural Metrics: BLEURT-20, XCOMETXL.
• Embedding Metrics: BERTSCORE, LUXEMBEDDER (cosine similarity).
• Surface Overlap: BLEU, CHRF2, TER (reported in appendix only, as they are less reliable for inter-system comparison).
• Quality Estimation (QE): LUXEMBEDDER was tested as a reference-less metric.

3. Key Contributions

1. LUXMT System: A specialized MT system based on GEMMA 3, fine-tuned for LB→FR and LB→EN.
2. Novel Benchmark: A human-translated, contamination-free benchmark derived from the Luci magazine, covering LB-FR, LB-EN, and LB-DE.
3. Data Filtering Methodology: Demonstration that LUXEMBEDDER is highly effective at filtering parallel corpora, significantly improving model performance even when discarding large portions of data.
4. Cross-Lingual Generalization: Evidence that fine-tuning on LB→FR/EN improves performance on LB→DE (German), despite German data being absent from the training set.
5. QE Metric Exploration: An initial investigation into using LUXEMBEDDER as a reference-less Quality Estimation metric.

4. Results

• Performance Gains: LUXMT showed substantial improvements over the GEMMA 3 baseline across all language pairs.
  • LB→FR: +1.9 points on XCOMETXL, +1.3 on BLEURT-20.
  • LB→EN: +1.2 on XCOMETXL, +0.9 on BLEURT-20.
  • LB→DE: +0.6 on XCOMETXL, +1.4 on BLEURT-20 (statistically significant improvements despite zero German training data).
• Filtering Impact: Using a LUXEMBEDDER threshold of 0.99 on the news corpus yielded the highest scores (BERTSCORE: 94.4, BLEURT-20: 78.8) compared to lower thresholds (0.90, 0.95).
• QE Metric Correlation: LUXEMBEDDER showed strong correlations with other metrics, particularly CHRF2 and surface-overlap metrics. However, the authors note this is surprising given LUXEMBEDDER's semantic nature, suggesting it should be used with caution.

5. Significance & Limitations

Significance:

• This work establishes a new state-of-the-art for Luxembourgish translation, proving that high-quality MT is achievable for low-resource languages using targeted fine-tuning and rigorous data filtering.
• It highlights the "zero-shot" cross-lingual capabilities of modern LLMs, where training on two target languages improves a third untrained language.

Limitations:

• Domain Specificity: The benchmark is limited to a tourist magazine, which may not reflect performance in other domains (e.g., legal or technical).
• Benchmark Size: 500 segments per pair is small, potentially leading to high variance in metric deltas.
• Translation Artifacts: The source language of the Luci magazine is unclear; if the Luxembourgish text was translated from another language, it may contain "translationese."
• QE Metric Utility: While LUXEMBEDDER correlates with other metrics, its utility as a standalone QE metric requires further validation against human preference data.

Future Work:
The authors plan to release linguistically-motivated test suites, fine-tune the new Translategemma model, and explore filtering based on estimated translation difficulty.