Syntax as a Rosetta Stone: Universal Dependencies for In-Context Coptic Translation

1. Problem Statement

The paper addresses the challenge of Machine Translation (MT) for Low-Resource Languages (LRLs), specifically Coptic (Sahidic dialect).

• The Gap: While Large Language Models (LLMs) have achieved high performance in high-resource languages, they fail significantly on LRLs like Coptic due to a lack of training data. Standard "out-of-the-box" prompting often yields fluent but semantically incorrect translations because models lack grammatical intuition for the language.
• Limitations of Existing Solutions: Previous approaches using in-context learning (ICL) with bilingual dictionaries (lexical augmentation) improve vocabulary accuracy but fail to capture complex grammatical structures, morphological nuances, and syntactic relations inherent in Coptic (an agglutinative, head-initial language).
• The Core Question: Can augmenting ICL prompts with syntactic information (specifically Universal Dependencies parses) significantly improve translation quality beyond what lexical dictionaries alone can achieve?

2. Methodology

The authors propose a novel In-Context Learning (ICL) framework that augments translation prompts with both lexical and syntactic data derived from Universal Dependencies (UD).

A. Data and Resources

• Source Language: Coptic (Sahidic dialect).
• Target Language: English.
• Datasets:
  • Coptic Scriptorium UD Treebank: ~60k tokens (2,387 sentences) with gold-standard UD parses and translations. Split into development (380 sentences) and test (405 sentences).
  • Ostraca: Out-of-domain data (21 sentences) from a previous study (Wannaz and Miyagawa, 2024).
• Lexicon: The Coptic Dictionary (Feder et al., 2018), containing over 10,000 entries with morphological and dialect-specific information.
• Parser: Coptic-NLP pipeline, which performs segmentation and generates UD parses (automatic parses with ~90% accuracy).

B. Prompt Augmentation Components

The authors designed four specific components to inject linguistic knowledge into the LLM prompt:

1. LEX (Lexicon): Retrieves dictionary entries for words in the source sentence. It filters for relevance, providing lemma, part-of-speech, and sense-level translations to guide vocabulary selection.
2. DEP (Dependency): Verbalizes syntactic dependency relations into plain English statements (e.g., "n is the case marking of yoyou"). This explicitly exposes head-dependent relationships that are not obvious from surface word order.
3. CON (Construction): A bespoke module that identifies specific, difficult grammatical constructions (26 patterns identified via error analysis) using DepEdit. It provides targeted instructions on how to translate these structures (e.g., handling postponed subjects or counterfactual conditionals) and transliterates proper nouns.
4. CoNLLU: Includes the raw CoNLL-U format output from the parser as a low-effort syntactic representation.

C. Experimental Setup

• Models Tested:
  • Open-Weight: Gemma3 (12B, 27B), Llama3.1 (8B), Aya-Expanse (8B, 32B).
  • Closed-Source: GPT-4.1 and GPT-4.1 Mini.
• Settings: The study compares:
  • Baseline (No augmentation).
  • Lexicon only (LEX).
  • Syntax only (CoNLLU, DEP, CON, DEP+CON).
  • Combined (LEX + SYNTAX, where SYN includes all syntactic components).
• Evaluation Metrics: Primary metric is BERTScore (Avg. F1) due to its correlation with human evaluation in LLM settings. BLEU and METEOR are reported for comparison with prior work, though BLEU is noted as unstable for small datasets.

3. Key Contributions

1. First Application of UD for Coptic MT: This is the first study to utilize Universal Dependencies parses as a source of grammatical information for machine translation of Coptic.
2. Syntactic Augmentation Framework: The paper introduces a modular approach to injecting syntactic knowledge into prompts via verbalized dependencies (DEP), targeted construction rules (CON), and raw parse data (CoNLLU).
3. Demonstration of Complementarity: The work proves that syntactic information is not merely redundant to lexical information; rather, it provides complementary benefits that significantly boost performance when combined with dictionaries.
4. Validation of Automatic Parsing: The study demonstrates that high-quality automatic parses (from Coptic-NLP) are nearly as effective as gold-standard parses for in-context translation, making the approach scalable without requiring manual treebanking for every new text.
5. State-of-the-Art Results: The proposed method achieves new state-of-the-art results for Coptic-to-English translation across multiple model sizes.

4. Results

• Performance Gains:
  • Lexicon (LEX) alone provides significant improvements over the baseline.
  • Syntax (SYN) alone offers gains, but the most substantial improvements occur when LEX and SYN are combined (LEX+SYN).
  • The LEX+SYN setting achieved the highest BERTScore across all models (e.g., Gemma-27B improved from 0.8375 to 0.8736; GPT-4.1 improved from 0.9012 to 0.9195).
• Model Sensitivity:
  • Larger Models (GPT-4.1, Gemma-27B): Benefit significantly from raw CoNLL-U data and verbalized dependencies.
  • Smaller Models (Gemma-12B): Benefit most from the CON (Construction) component, which provides explicit, simplified instructions on difficult grammatical patterns.
• Automatic vs. Gold Parses: There was no statistically significant difference in translation quality between using automatic parses and gold-standard parses, suggesting that parser errors (~10% error rate) do not cascade into catastrophic translation failures in this context.
• Domain Specificity: Models performed better on Biblical texts than non-Biblical texts, likely due to memorization of Bible verses by LLMs. However, the augmentation strategy improved performance in both domains.
• Out-of-Domain: On the Ostraca dataset, the LEX+SYN setting outperformed previous reported results (including Claude Opus) on METEOR scores.

5. Significance and Conclusion

This paper establishes that syntactic information acts as a "Rosetta Stone" for low-resource languages, bridging the gap between raw vocabulary and grammatical structure.

• Practical Impact: The findings suggest that for endangered or ancient languages where training data is scarce, researchers can achieve high-quality translation by combining existing dictionaries with automated syntactic parsing, rather than relying solely on massive parallel corpora.
• Methodological Shift: It moves the field beyond simple "dictionary-in-prompt" strategies toward structured linguistic augmentation, proving that LLMs can effectively utilize explicit grammatical rules when provided in the context.
• Future Directions: The authors advocate for extending this approach to other semantic resources (e.g., WordNet, semantic graphs) and applying it to other low-resource languages with available UD treebanks.

In summary, the paper demonstrates that combining lexical glosses with syntactic analysis is the most effective strategy for in-context machine translation of Coptic, setting a new benchmark for low-resource language processing.