``Don't Teach Minerva'': Evaluating Retrieval-Augmented Refinement for Low-Resource Latin Translation with New Benchmarks

Technical Summary: "Don't Teach Minerva": Evaluating Retrieval-Augmented Refinement for Low-Resource Latin Translation

Problem Statement
Machine translation (MT) for Latin faces unique challenges due to its morphological richness, flexible word order, and diachronic variation spanning over two millennia. These linguistic complexities are compounded by severe resource scarcity: high-quality parallel corpora are limited, often stylistically narrow, and frequently rely on archaic 19th-century English reference translations that penalize modern surface-level metrics (e.g., BLEU) even when semantic meaning is preserved. Furthermore, historical translation often admits multiple legitimate interpretations, requiring a balance between semantic adequacy and structural fidelity that standard MT systems struggle to achieve. While recent proprietary Large Language Models (LLMs) have improved zero-shot performance, they introduce issues regarding reproducibility, cost, and lack of auditability, which are critical for scholarly workflows.

Methodology
The paper proposes a fully reproducible, open-source, two-stage pipeline for Latin-to-English translation that avoids task-specific fine-tuning of the final refinement model. The architecture consists of:

1. Specialized Drafter (Stage 1): A fine-tuned NLLB-200-1.3B model (encoder-decoder) generates an initial translation draft. This model is adapted through a two-phase protocol:

   • Denoising Adaptation: Trained on a 100M-token monolingual Latin corpus with synthetic noise to internalize historical linguistic features.
   • Parallel Fine-tuning: Trained on a harmonized corpus of ~160k Latin-English pairs (combining Rosenthal's classical/biblical data and OPUS Bible).
   • Role: This draft serves as a structurally conservative anchor, preserving clause structure, polarity, and case-governed roles.

2. Zero-Shot Refiner with Retrieval-Augmented Generation (Stage 2): A decoder-only LLM (Llama-3.3-70B or Qwen3 variants) refines the draft in a zero-shot setting. The refinement process is augmented via Retrieval-Augmented Generation (RAG):

   • Retrieval: Semantically similar Latin source segments are retrieved from a large non-parallel Latin corpus (100M tokens) using dense embedding (BAAI/bge-m3).
   • Context Construction: The system generates English drafts for these retrieved segments using the same NLLB drafter.
   • Prompting: The refiner receives the original Latin text, the NLLB draft, and up to five retrieved source-draft exemplar pairs as in-context guidance.

Key Contributions

• New Resources: The authors introduce a task-ready training corpus (harmonized classical and biblical parallel data) and a new out-of-domain (OOD) evaluation set consisting of 12th-century letters by Yves de Chartres, evaluated against modern scholarly translations.
• Reproducible Pipeline: A transparent, open-source workflow that combines a specialized NMT drafter with zero-shot LLM refinement, eliminating the need for proprietary black-box models.
• Evaluation Framework: A rigorous assessment using both in-domain (Rosenthal) and OOD benchmarks, employing lexical metrics (BLEU, chrF++, METEOR) and semantic metrics (BERTScore, COMET) to capture the nuances of historical translation.

Results

• In-Domain Performance: On the standard Rosenthal test set, the open-source RAG system (specifically Llama-3.3-70B + RAG) achieved performance comparable to, and in some metrics exceeding, a strong proprietary baseline (GPT-5.1). The system reached a COMET score of 0.743 compared to GPT-5.1's 0.730.
• Out-of-Domain Performance: On the Yves de Chartres OOD set, the pipeline demonstrated robust generalization. The draft conditioning alone provided the majority of semantic gains (increasing COMET from 0.714 to 0.742), while the addition of retrieved exemplars further improved lexical fidelity (BLEU increased from 30.28 to 34.28).
• Ablation Studies: Fine-tuning the refiner models on a small subset of the parallel corpus improved in-domain results but led to overfitting, causing performance drops on the OOD benchmark. This confirmed that inference-time guidance (RAG) is more effective for cross-domain robustness than direct model adaptation.
• Qualitative Analysis: Case studies revealed that the RAG pipeline effectively resolved complex pronoun linkages, preserved passive voice and case-governed relations, and disambiguated formulaic lexical choices where zero-shot models failed.

Significance and Claims
The paper claims that for low-resource, morphologically rich historical languages, high-quality translation does not require task-specific fine-tuning of the refiner. Instead, a "division of labor" approach—where a specialized drafter provides structural anchoring and retrieval provides analogical guidance—yields results competitive with proprietary models.

The authors emphasize that the pipeline offers a distinct translation profile: while proprietary models often prioritize idiomatic fluency, this open-source approach preserves the "visible scaffolding" of the Latin source text. This structural transparency makes the outputs more auditable and traceable, which is a critical advantage for scholarly, pedagogical, and legal applications. The study concludes that structured inference-time guidance is a more viable strategy for historical MT than narrow model adaptation, offering a practical balance between adequacy, reproducibility, and philological traceability.