Open Machine Translation for Esperanto

1. Problem Statement

Esperanto, the most widely used constructed language, possesses a significant online presence and substantial textual resources (e.g., Wikipedia, Common Crawl). However, it remains under-resourced in the context of modern Natural Language Processing (NLP) and Machine Translation (MT).

• The Gap: Despite its digital footprint, there is no state-of-the-art, openly available MT system specifically optimized for Esperanto. Commercial platforms exist but are closed-source.
• The Challenge: While Esperanto has regular grammar and productive word formation, its representation in large-scale pretraining corpora is often insufficient for specialized tasks, and dedicated evaluation benchmarks are scarce.
• Goal: To establish the first comprehensive benchmark of open-source MT systems for Esperanto and develop efficient, open models to support the language's community values of linguistic equality and decentralization.

2. Methodology

The authors adopted a two-pronged approach: Benchmarking existing systems and Developing new models.

A. Experimental Setup

• Language Pairs: Translation between Esperanto (eo) and three major languages: English (en), Spanish (es), and Catalan (ca). Both directions (eo $\leftrightarrow$ target) were evaluated.
• Data:
  • Training: Used the Tatoeba Challenge (OPUS repository) data. After filtering (length, ratio, HTML removal, language ID), the dataset ranged from ~673k to 45M sentence pairs depending on the language pair.
  • Evaluation: Used the FLORES+ benchmark, which provides professionally translated, multi-domain test sets.
• Metrics:
  • Automatic: BLEU, ChrF++ (primary metric due to better correlation with human judgment for morphologically rich languages), COMET, and MetricX.
  • Human: Pairwise ranking of the top three models per architecture by human annotators (native Esperanto speakers).

B. Models Evaluated

1. Rule-Based: Apertium (manually crafted rules/dictionaries).
2. Neural Encoder-Decoder (Pre-trained): The NLLB (No Language Left Behind) family, including models from 600M to 3.3B parameters and distilled variants.
3. General-Purpose LLMs: Llama-3.1-8B-Instruct (zero-shot prompting).
4. MT-Tuned LLMs: TowerInstruct-7B-v0.2 and Tower-Plus-9B (models fine-tuned specifically for translation tasks).

C. Models Developed (Training)

The authors trained their own models to fill the gap:

1. Compact Transformers: Trained from scratch using Marian.
   • Transformer-base: 60.6M parameters.
   • Transformer-tiny: 17.4M parameters.
2. Fine-Tuned LLM: Supervised fine-tuning of Llama-3.1-8B-Instruct using LoRA (Rank 16, Alpha 32) with instruction-style prompting.

3. Key Contributions

1. First Systematic Benchmark: Established the first standardized evaluation framework for open-source Esperanto MT, covering rule-based, encoder-decoder, and LLM architectures.
2. Open Release: Released all code (GitHub) and best-performing models (HuggingFace) to support the open-source Esperanto community.
3. Efficiency Analysis: Demonstrated that compact, task-specific models (17M–60M parameters) can achieve performance comparable to much larger general-purpose LLMs (8B parameters), offering a sustainable alternative for low-resource deployment.
4. Human Evaluation & Error Analysis: Provided qualitative insights into error patterns (lexical, grammatical, hallucination) and validated automatic metrics against human judgment.

4. Key Results

A. Performance Rankings (ChrF++)

• Best Overall: The NLLB family consistently outperformed all other systems. The NLLB-200-3.3B model achieved the highest scores, followed closely by the distilled 1.3B variant.
• LLM Performance:
  • General-purpose LLMs (Llama-3.1-8B): Performed reasonably well, often surpassing rule-based systems but lagging behind specialized NLLB models.
  • MT-Tuned LLMs (Tower): Surprisingly underperformed. TowerInstruct-7B-v0.2 failed significantly in several directions (e.g., generating non-Esperanto output), suggesting that fine-tuning on high-resource languages can degrade performance on low-resource languages.
• Trained Compact Models:
  • The Transformer-base (60.6M) model performed competitively with the fine-tuned Llama-3.1-8B, surpassing it in 4 out of 6 language directions.
  • The Transformer-tiny (17.4M) model achieved surprisingly strong results, proving that small architectures are viable for low-resource MT.

B. Asymmetry

Translating from Esperanto generally yielded higher scores than translating into Esperanto. This suggests that while models can understand Esperanto (due to pretraining), generating it remains more challenging due to data scarcity.

C. Metric Correlation

• Neural Metrics: MetricX and COMET showed the strongest correlation with human judgments, even though they were not explicitly fine-tuned on Esperanto.
• Traditional Metrics: BLEU showed weak correlation (near random in some directions), while ChrF++ showed moderate correlation. This reinforces the shift toward learned metrics for low-resource evaluation.

D. Qualitative Error Analysis

• NLLB: Produced fluent output but occasionally made "compositional" errors (e.g., translating neĝtabulo literally as tabla de nieve instead of the conventional snowboard).
• Llama-FT: Prone to hallucinations, adding/omitting information, and mixing languages.
• Transformer-base: Produced accurate but literal translations, sometimes failing with named entities or complex grammar.

5. Significance and Implications

• Practical Recommendation: For Esperanto and similar low-resource languages, NLLB should be the default choice if computational resources allow. If resources are constrained, compact Transformer models trained from scratch are a highly efficient alternative to massive LLMs.
• LLM Tuning Caution: The study warns that fine-tuning general-purpose LLMs on specific high-resource languages (like Tower) may negatively impact their ability to translate low-resource languages. General-purpose LLMs often remain more robust than specialized MT-tuned LLMs in these scenarios.
• Sustainability: The success of 17M–60M parameter models aligns with the Esperanto community's ethos of accessibility. These models can run on standard CPUs and personal devices, lowering the barrier to entry for language technology.
• Future Directions: The authors suggest exploring Esperanto as a pivot language (interlingua) for multilingual MT and investigating hybrid approaches combining rule-based (Apertium) and neural methods.

In conclusion, this paper provides a foundational step toward democratizing Esperanto language technology, proving that high-quality, open, and efficient machine translation is achievable without relying on massive, closed-source models.