💬 NLP

Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

This paper evaluates large language models' ability to perform in-context translation using synchronous context-free grammars as a formal proxy for natural languages, finding that performance significantly degrades with increased grammar size and sentence length, is hindered by morphological and script differences, and is characterized by errors such as vocabulary hallucinations and untranslated source words.

Original authors: Jackson Petty, Jaulie Goe, Tal Linzen

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jackson Petty, Jaulie Goe, Tal Linzen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced translator how to speak a language they have never heard before. You can't give them a dictionary or a textbook they've studied for years. Instead, you hand them a brand-new, rulebook written in a foreign language, along with a few sentences, and say, "Here are the rules. Now, translate this sentence."

This is the core challenge of In-Context Machine Translation (ICMT). The big question researchers asked was: Can modern AI models (like GPT-5 or Gemma) actually learn a new language just by reading its grammar rules on the fly, without needing massive amounts of training data?

To answer this, the authors created a "training camp" using formal languages (made-up languages with strict, logical rules) instead of real-world languages like French or Swahili. This allowed them to control every variable perfectly.

Here is the breakdown of their findings, using some everyday analogies:

1. The "Rulebook Size" Problem

The Finding: The bigger the grammar rulebook, the worse the AI performed.
The Analogy: Imagine you are trying to assemble a piece of IKEA furniture.

Small Grammar: The rulebook is a simple 2-page pamphlet for a small stool. The AI can read it, understand the steps, and build the stool perfectly.
Large Grammar: The rulebook is a 500-page manual for a massive, complex castle. The AI gets overwhelmed. It starts forgetting the first rule by the time it reaches the last one.
Result: When the grammar was small (a few hundred rules), the AI was nearly perfect. But as the grammar grew to the size needed to model human language (thousands of rules), the AI's accuracy plummeted. It simply couldn't keep all the rules in its "working memory" at once.

2. The "Sentence Length" Problem

The Finding: Short sentences were easy; long sentences were a disaster.
The Analogy: Think of it like a game of "Telephone."

Short Sentence: You whisper a 3-word phrase to the AI. It translates it perfectly.
Long Sentence: You whisper a 30-word story. By the time the AI gets to the end of the sentence, it has forgotten the beginning. It starts hallucinating words or dropping parts of the sentence entirely.
Result: The AI handled sentences under 10 words well, but once they passed 20 words, performance dropped sharply.

3. The "Word Order" Surprise

The Finding: Changing the order of words (like putting the verb at the end of the sentence) didn't actually hurt the AI much.
The Analogy: Imagine you are rearranging furniture in a room.

If you tell the AI, "Put the chair in front of the table," it can easily swap that to "Put the table behind the chair" if the rulebook says so.
Result: The AI is actually quite good at following instructions to shuffle word order (e.g., English style vs. Japanese style). The difficulty wasn't the order; it was the complexity of the rules themselves.

4. The "Morphology" Trap

The Finding: The AI struggled most when translating from a "simple" language (where words don't change) to a "complex" language (where words change based on who is doing the action).
The Analogy: Imagine a game where you have to guess a secret code.

Simple to Simple: "Cat" always means "Cat." Easy.
Simple to Complex: In the target language, "Cat" changes to "Cat-er" if a man is looking at it, and "Cat-est" if a woman is looking at it. The AI has to look at the other words in the sentence to figure out which version of "Cat" to use.
Result: The AI was great at simple-to-simple translation but terrible at figuring out the hidden "agreement" rules when moving from a simple language to a complex one.

5. The "Script" Nightmare

The Finding: The AI did great with Latin letters (A, B, C), okay with Cyrillic (Russian style), but failed completely with Hebrew, especially if it had vowel marks.
The Analogy: Imagine the AI is a chef who has only ever cooked with a standard set of knives.

Latin Script: Standard knives. No problem.
Cyrillic/Hebrew: The chef has to use a weird, unfamiliar set of tools. They can still chop, but they are clumsy.
Hebrew with Vowels: This is like being handed a set of tools made of glass that you've never seen before. The AI completely froze. It couldn't even produce the correct characters.
Result: The AI relies heavily on patterns it has seen before. If the "look" of the words (the script) is too unfamiliar, the AI gives up, even if it understands the grammar rules perfectly.

The Big Takeaway

The paper concludes that while AI models can technically learn a new language from a rulebook, they are extremely fragile at it.

They are like a student who can ace a math test if the problems are short and the formulas are simple.
But if you give them a 50-page textbook of complex formulas and ask them to solve a long, multi-step problem in a language they've never seen, they will likely panic, forget the rules, and start making things up.

Why does this matter?
Currently, AI translation relies on massive amounts of data (millions of sentences) to learn low-resource languages. This study suggests that simply handing an AI a grammar book isn't enough yet. Until AI gets better at holding complex rules in its "mind" and recognizing unfamiliar writing systems, we can't fully rely on it to translate languages it hasn't been trained on.

1. Problem Statement

Low-resource languages pose a significant challenge for Large Language Models (LLMs) because standard neural machine translation (NMT) systems rely heavily on massive parallel corpora, which do not exist for many languages. A proposed solution is In-Context Machine Translation (ICMT), where an LLM is provided with linguistic descriptions (e.g., reference grammars, dictionaries) in its context window at inference time to translate into a language it has never seen during training.

However, evaluating ICMT on natural languages is difficult due to:

Ambiguity: Natural language sentences can be translated in multiple valid ways, making evaluation reliant on string-overlap metrics (like BLEU) which often overestimate quality.
Confounding Factors: It is hard to isolate whether a model is using the provided grammar or relying on biases from its pre-training data (e.g., preferring common word orders or scripts).
Previous Findings: Recent studies (e.g., Aycock et al., 2024) suggest LLMs rely more on few-shot examples than grammatical descriptions, casting doubt on the viability of pure grammar-based ICMT.

2. Methodology

To address these limitations, the authors propose a formal analogue of ICMT using Synchronous Context-Free Grammars (SCFGs).

Formal Framework: They construct SCFGs that define pairs of formal languages (Source $L_s$ and Target $L_t$ ). Each production rule in the grammar is a pair: $A \to \langle \alpha, \beta \rangle$ , where $\alpha$ is a string in $L_s$ and $\beta$ is the corresponding string in $L_t$ .
Controlled Variables: The authors generate novel, naturalistic languages with no overlap with LLM training data. They systematically vary five key parameters:
1. Grammar Size ( $|G|$ ): Ranging from ~25 to 10,000 production rules.
2. Sentence Length ( $\ell$ ): Ranging from 3 to 50 words.
3. Word Order: Testing Subject-Verb-Object (SVO), Subject-Object-Verb (SOV), and Object-Verb-Subject (OVS) permutations.
4. Morphology: Varying the presence of person/number agreement (Agreement vs. No Agreement) in verbal morphology.
5. Orthography: Testing Latin, Latin with diacritics, Cyrillic, Hebrew (without vowels), and Hebrew (with vowel pointing/nikkud).
Models Evaluated: OpenAI's GPT-5 series (nano, mini, base) and Google's Gemma 3 series (1B, 4B, 12B).
Evaluation Metric: The primary metric is Exact Match Accuracy (comparing the model's output $\hat{t}$ against the gold translation $t$ derived deterministically from the SCFG). Secondary metrics include Bag-of-Words, BLEU, and chrF++.

3. Key Contributions

Formal Abstraction of ICMT: The paper introduces a rigorous, controllable framework using SCFGs to isolate the specific cognitive skill of translating based on in-context grammatical rules, removing the noise of natural language ambiguity and training data contamination.
Systematic Benchmarking: It provides the first comprehensive evaluation of how grammar complexity, sentence length, morphological complexity, and script familiarity impact LLM translation performance in a zero-shot, grammar-only setting.
Error Taxonomy: The authors categorize failure modes into specific types: Recall errors (wrong word from target vocab), Source vocabulary leakage (copying source words), Omission, Hallucination (inventing words), and Orthography errors (wrong script).

4. Key Results

The experiments yielded five major findings:

Grammar Size and Sentence Length: Performance drops sharply as complexity increases.
- Models achieve near-perfect accuracy on small grammars (<100 rules) and short sentences (<10 words).
- Accuracy collapses as grammar size approaches human-language scales (several thousand rules) and sentence lengths exceed 20 words. GPT-5-mini and GPT-5-nano fail almost completely on large grammars, while even GPT-5 (base) sees a significant decline.
Word Order: Surprisingly, differences in word order (e.g., translating SVO to SOV or OVS) have negligible impact on performance. Models can successfully reorder words based on the provided rules, suggesting that syntactic reordering is not the primary bottleneck.
Morphology: Models struggle significantly with agreement.
- Translation is easiest between languages with no agreement morphology.
- Translation is hardest when moving from a language without agreement to one with agreement (NoAgr $\to$ Agr), as the model must infer specific morphological features not present in the source.
Orthography (Script): This is a critical bottleneck.
- Performance is highest for Latin scripts.
- It drops for Cyrillic and Hebrew.
- Total Failure: Models achieve 0% exact-match accuracy when translating into Hebrew with vowel pointing (nikkud), regardless of grammar size or sentence length. This suggests models cannot reliably generate unfamiliar character sequences even when the rules are explicit.
Metric Discrepancy: Standard heuristic metrics (BLEU, chrF++) consistently overestimate model performance, particularly as tasks become harder. Exact match accuracy is a much stricter and more reliable indicator of true capability.

5. Significance and Discussion

Capability vs. Robustness: The results confirm that LLMs can use in-context grammatical descriptions to translate, contradicting claims that they rely solely on examples. However, their robustness is severely limited by the complexity of the rule set and the unfamiliarity of the script.
The "N-gram" Bottleneck: The failure on complex grammars and unfamiliar scripts (like Hebrew with vowels) suggests that LLMs are not purely performing abstract rule induction. Instead, they rely on generating token sequences that have high n-gram similarity to their training data. When the target script or rule structure deviates too far from training distributions, the model fails to "copy" or "emit" the correct sequence, even if it understands the logic.
Implications for Low-Resource Languages: While ICMT offers a theoretical path to translating low-resource languages using reference grammars, current models are not yet robust enough to handle the full complexity of natural language grammar and diverse writing systems without parallel data or few-shot examples.
Future Directions: The study highlights the need for models that can better decouple abstract rule application from token-level distributional biases, particularly for non-Latin scripts and morphologically rich languages.

In conclusion, the paper demonstrates that while LLMs possess the potential for grammar-based translation, their current performance is brittle, heavily dependent on the simplicity of the task and the familiarity of the writing system.