💬 NLP

A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

This paper presents a pipeline that leverages large language models to extract grammatical rules and examples from descriptive grammar books to generate synthetic parallel corpora, demonstrating that fine-tuning machine translation models on this synthetic data significantly improves performance for endangered languages like Kalamang, Tuatschin, and Mandan compared to seed-data baselines.

Original authors: Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger, Rico Sennrich

Published 2026-07-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger, Rico Sennrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling library where most books are written in languages like English, Spanish, or Mandarin. These languages have millions of copies, endless stories, and huge crowds of people reading and writing them every day. But deep in the dusty, forgotten corners of this library, there are thousands of other languages—some spoken by only a few hundred people—that are slowly fading away. These are "endangered languages." For decades, computers have struggled to understand or translate these languages because they lack the massive piles of "parallel data" (pairs of sentences in two languages) needed to teach them. Without these pairs, the computer is like a student trying to learn a new language with no textbooks, no teachers, and no practice sentences.

However, these languages aren't entirely silent. Linguists have spent years visiting communities, listening to stories, and writing down "grammar books." These books are like detailed instruction manuals for how a language works, filled with rules about how words change, how sentences are built, and lists of vocabulary. The big question researchers have been asking is: Can we use these old, static grammar books to teach a computer how to translate, even if we don't have the modern piles of practice sentences? It's a bit like trying to teach someone to drive a car using only a manual, without ever letting them sit behind the wheel.

This paper, titled "A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books," takes a clever, semi-automated approach to solve this puzzle. The authors, a team from the University of Zurich, built a digital pipeline that acts like a very strict, rule-following robot. Instead of just asking a super-smart AI to "guess" the translation based on the grammar book (which often leads to confusion), they first extract the rules, word lists, and example sentences from the PDFs of these grammar books. Then, they use a Large Language Model (LLM) to act as a creative but disciplined writer. This writer takes an existing sentence from the grammar book, swaps out a word (like changing "big" to "small"), and then strictly applies the grammar rules to make sure the new sentence still makes sense grammatically.

The result is a "synthetic corpus"—a brand new set of practice sentences generated entirely by the computer, based on the rules in the book. They tested this method on three very different, extremely low-resource languages: Kalamang (from Papua New Guinea), Tuatschin (a dialect in Switzerland), and Mandan (from North America). They didn't just guess; they ran a massive "factorial study," which is a fancy way of saying they tried every possible combination of settings. They changed the type of words they swapped (nouns, verbs, adjectives), how much of the grammar book they looked at for clues (just the specific rule or the whole chapter), and how many new sentences they generated (from 5 to 20 per original sentence).

The findings are a mix of exciting success and important caution. For Kalamang, the method worked beautifully: in 75% of the different settings they tried, the computer learned better than it did with just the tiny amount of real human data available. In the best-case scenario, the translation quality jumped by a huge margin (an improvement of +8.8 points on their scoring scale). For Tuatschin, it also worked well, but only if they generated enough sentences; if they tried to learn with too few examples, the computer got confused and performed worse. However, for Mandan, a language with incredibly complex sentence structures, the method struggled. The computer often learned the wrong things, producing sentences that looked grammatically correct but made no sense, or it simply failed to improve over the baseline.

The paper explicitly rules out the idea that you can just feed the entire grammar book into an AI and expect it to instantly become a perfect translator. They found that while reading the whole book helps in some cases, it's often better to give the AI specific, codified rules to follow when generating new practice sentences. They also argue against the idea that "more data is always better." In the case of Mandan, generating too many synthetic sentences actually made the computer worse, because it started memorizing the weird patterns of the generated data instead of learning how to translate.

Ultimately, this research suggests that we can repurpose old, static grammar books into dynamic training tools for computers. It's not a magic wand that solves the problem for every language immediately, but it offers a practical, scalable path forward. By turning linguistic manuals into synthetic practice drills, we might finally be able to build translation tools for the world's most endangered languages, giving them a digital voice before they are lost forever. The authors are careful to note that this is a starting point, not a finished product, and that human speakers will still need to validate the results, but it's a significant step toward keeping these languages alive in the digital age.

Technical Summary: A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

Problem Statement

Despite the existence of descriptive grammar books for thousands of endangered languages, these resources remain largely unexploited for Natural Language Processing (NLP). Most endangered languages lack the parallel corpora required to train effective Machine Translation (MT) systems. While Large Language Models (LLMs) have shown promise in low-resource settings, prior approaches face significant bottlenecks:

Context Injection Costs: Prompting LLMs with entire grammar books at inference time is computationally expensive and often yields diminishing returns, as models tend to learn from embedded parallel examples rather than narrative grammatical descriptions (Aycock et al., 2025).
Manual Effort: Converting grammatical rules into structured, executable representations typically requires expert linguistic annotation, which does not scale to the thousands of languages with existing grammars.
Data Scarcity: Existing synthetic data generation methods often rely on monolingual language model probabilities or unconstrained generation, lacking the morphological precision necessary for languages with complex structures.

The central question addressed is whether the structured content of grammar books can be computationally operationalized to generate novel, high-quality synthetic parallel sentences for fine-tuning MT models, rather than merely using them as context during inference.

Methodology

The authors introduce a semi-automated pipeline that extracts linguistic information from unstructured PDF grammar books and synthesizes parallel corpora via rule-guided lexical substitution. The process involves three stages:

1. Linguistic Information Extraction

Parallel Sentence Extraction: A two-step process converts PDF text into lines, followed by a fine-tuned BERT classifier (trained on ~200 annotated lines per language) to identify source, gloss, and translation lines. Rule-based post-corrections ensure triplet reconstruction.
Section and Dictionary Extraction: Heuristic algorithms detect section headers based on font size statistics. Non-grammatical sections are filtered using embedding-based semantic similarity against a taxonomy of linguistic categories. Word lists are semi-automatically extracted from glossary sections and annotated with part-of-speech (POS) tags using spaCy.
Rule Extraction and Codification: Narrative grammatical descriptions are converted into codified rules using Gemini-2.5-flash. The model extracts rules conforming to the UniMorph annotation schema (structured YAML objects containing target POS, affix type, feature-value pairs, and context dependencies). These are then converted into deterministic pseudo-code functions (ApplyRule(STEM, POS)) to ensure unambiguous application.

2. Synthetic Corpus Construction

The pipeline generates synthetic parallel sentences by replacing content words in extracted "seed" sentences with dictionary entries, guided by retrieved grammatical rules. The study employs a fully-crossed factorial design varying three factors:

Target Part-of-Speech (POS): Noun, Verb, Adjective, or Adverb.
Retrieval Granularity:
- Rule-level: Retrieval of specific codified rules and their textual descriptions.
- Section-level: Retrieval of full source sections from the grammar book to preserve phrasal context.
Sample Volume ( $k$ ): The number of synthetic variants generated per seed sentence ( $k \in \{5, 10, 15, 20\}$ ).

3. Evaluation

The generated corpora are used to fine-tune Gemini-2.5-flash (using LoRA adapters). Performance is evaluated against baselines including zero-shot LLMs, NLLB-200 with proxy languages, and Seed-Data Fine-Tuning (SDFT) using only the original extracted parallel sentences. Evaluation metrics include BLEU, ChrF, and ChrF++.

Key Contributions

Automated Pipeline: The first semi-automated framework to extract linguistic information from grammar books and synthesize parallel corpora with minimal human intervention (requiring only ~200 annotated lines for the classifier).
Factorial Study: A systematic evaluation across 96 configurations (3 languages $\times$ 4 POS $\times$ 2 granularities $\times$ 4 sample volumes) to identify which factor combinations drive translation quality gains.
Operationalization of Grammar Books: Demonstrating that static linguistic documentation can be repurposed for MT fine-tuning, moving beyond simple context injection to active data generation.

Results

The study validates the pipeline on three typologically diverse languages: Kalamang (Papuan, agglutinative), Tuatschin (Romance, analytic), and Mandan (Siouan, polysynthetic).

Overall Performance:
- Kalamang: Synthetic augmentation improved over the SDFT baseline in 75% of configurations, with a best-case ChrF++ gain of +8.8.
- Tuatschin: Improvements were observed in 59% of configurations, with a best-case gain of +5.3.
- Mandan: The language proved most resistant; only 16% of configurations exceeded the baseline, with a best-case gain of +3.3.
Factor Analysis:
- POS: Nouns provided the most consistent gains for Kalamang, while verbs were most effective for Tuatschin. No POS category reliably improved Mandan.
- Granularity: Section-level retrieval yielded higher gains for Kalamang, whereas rule-level retrieval was more effective for Tuatschin verbs.
- Sample Volume: Performance was non-monotonic. For Tuatschin, small volumes ( $k=5$ ) caused catastrophic failures, while performance recovered at $k \ge 10$ . For Mandan, excessive volume ( $k=20$ ) led to sharp degradation, where models began generating grammatical annotations rather than fluent translations.
Qualitative Findings: Synthetic training enabled correct syntactic reordering (e.g., Kalamang SOV to English SVO) and better handling of complex auxiliary constructions (Tuatschin). However, failure modes included literal translations of compound words and hallucinations under lexical uncertainty.

Significance and Limitations

The paper claims that this approach offers a practical path toward building translation tools for severely under-resourced languages that possess grammar books but lack parallel corpora. By producing standalone synthetic parallel corpora, the method decouples the data generation from specific model architectures, allowing the data to train smaller, locally deployable systems.

The authors remain modest regarding the generalizability of the results:

Evaluation: The study relies entirely on automatic metrics; the absence of native speaker evaluation prevents distinguishing genuine translation competence from test-set overfitting.
Scalability: The pipeline depends on English-specific NLP tools (spaCy) for POS tagging and rule extraction, limiting direct application to non-English target languages without adaptation.
Semantic Plausibility: The lexical substitution mechanism lacks semantic co-occurrence constraints, occasionally producing syntactically valid but semantically implausible sentences.
Scope: The pipeline was validated on only three languages; its performance on tonal systems, non-concatenative morphology, or non-standard grammar book formats remains untested.

The work does not claim to replace community-driven language revitalization efforts but aims to provide baseline translation tools that support documentation and education.