💬 NLP

Training Models on Dialects of Translationese Shows How Lexical Diversity and Source-Target Syntactic Similarity Shape Learning

This paper demonstrates that training small English language models on machine-translated data reveals that while general perplexity is primarily driven by the lexical diversity of the translated corpus, grammatical performance is strongly shaped by the typological similarity between the source language and English.

Original authors: Jenny Kunz

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jenny Kunz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to speak perfect English. Ideally, you'd feed it millions of books, news articles, and conversations written by native English speakers. But what if you don't have enough native text? What if you have to use books that were originally written in French, Japanese, or Swahili and then translated into English?

This is the problem Jenny Kunz tackled in her paper. She asked: Does it matter which language we translate from? And does the "flavor" of that translation change how the robot learns?

Here is the story of her findings, explained with some everyday analogies.

The "Accent" of Translation

First, the paper introduces a concept called Translationese. Think of native English text as a clear, crisp recording of a singer. Translationese is like a cover song. It's the same song, but the singer has a slight accent, maybe they sing the notes a little differently, or they skip a few complex harmonies.

Even professional human translators (and definitely AI translators) leave traces of their original language behind. A sentence translated from German might sound a bit stiff or structured differently than one written by a native speaker.

The Experiment: 24 Different Accents

Kunz took 24 different languages (ranging from Germanic cousins like Swedish to distant relatives like Swahili and Tamil) and translated huge amounts of text into English. She then trained small AI models on these "translated English" datasets.

She wanted to see two things:

How well does the robot predict the next word? (Like a autocomplete feature).
Does the robot actually understand grammar? (Like knowing the difference between "The cat is sleeping" and "The cat are sleeping").

The Big Discoveries

1. The "Vocabulary Buffet" vs. The "Grammar Gym"

The paper found that the type of language you translate from affects the robot differently depending on what you are testing.

For General Fluency (The Vocabulary Buffet):
When the robot is just trying to guess the next word in a sentence, the variety of words in the translated text matters most.
- Analogy: Imagine you are learning to cook. If you are given a cookbook translated from a language with a huge, diverse vocabulary (like a massive buffet), your robot learns to predict ingredients well, even if the grammar is a bit weird. It doesn't matter if the book came from a language similar to English; what matters is that the book had lots of different words to learn from.
- Result: In small data sets, the "richness" of the translated text was the biggest predictor of success.
For Grammar (The Grammar Gym):
When the robot needs to learn complex rules (like subject-verb agreement or long-distance connections in a sentence), how similar the source language is to English becomes the hero.
- Analogy: Imagine teaching someone to play tennis. If you teach them using a racket that looks and feels exactly like a standard tennis racket (a language structurally similar to English, like Swedish), they learn the proper swing faster. If you teach them with a racket that feels totally different (a distant language like Finnish), they struggle to learn the specific mechanics, even if you give them more practice time.
- Result: Once the robot had enough data, models trained on languages similar to English (like Swedish or Dutch) became much better at grammar than those trained on distant languages.

2. The "Flattening" Effect

No matter which language was used, the translated text always made the robot slightly "worse" than if it had learned from native English.

Analogy: It's like listening to a song played on a cheap speaker versus a high-end stereo. The cheap speaker (translated text) flattens the sound. It loses some of the nuance, the "sparkle," and the natural flow of the original. The robot trained on translations is always a bit less "native" than one trained on real English.

3. The "Family Reunion" Effect

The most fascinating finding was about how these robots interact with each other.

Analogy: Imagine two people who learned English by translating from two different languages. If those two source languages are "cousins" (structurally similar), the two robots understand each other's "accents" perfectly. If the source languages are strangers, the robots get confused.
Result: A robot trained on Swedish-translated text could easily understand text translated from Norwegian. But it struggled with text translated from a distant language like Arabic. This proves that translation creates specific "dialects" of English, and those dialects are shaped by the original language.

The Bottom Line

If you are building an AI and you have to use translated data:

If you just want the AI to sound okay generally: Pick a source language that has a rich, diverse vocabulary.
If you need the AI to be grammatically perfect: Pick a source language that is structurally similar to English (like other European languages).
The Reality Check: Translated data is a great backup plan when native text is scarce, but it will always leave the AI with a slight "accent" and a slightly weaker grasp of complex grammar compared to a native speaker.

In short: Translation is a bridge, but it's not the same as walking on the native soil. The closer the bridge is to the native soil (typological similarity), the better the robot learns the rules of the road.

1. Problem Statement

Machine-translated (MT) data is widely used in multilingual Natural Language Processing (NLP), especially for low-resource languages where native corpora are scarce. However, translated text exhibits systematic differences from native text, a phenomenon known as translationese. These differences include traces of the source language structure and characteristic stylistic patterns.

While previous work has established that translationese exists and can degrade model performance compared to native data, it remains unclear how the specific typological properties of the source language influence what a model learns. Specifically, does training on translations from typologically distant languages (e.g., Swahili) yield different linguistic competencies than training on translations from closely related languages (e.g., Swedish)? This paper investigates how the "dialect" of translationese shapes the linguistic knowledge of small language models.

2. Methodology

Experimental Setup

Models: The authors trained a suite of small, monolingual GPT-style Transformer models (125M parameters, 512-token context) following the Goldfish setup.
Data Budgets: Models were trained on two fixed data scales: 100MB and 1000MB.
Source Data: The study utilized the FineTranslations dataset, which contains machine translations of the FineWeb-2 corpus into English.
Source Languages: 24 typologically and genealogically diverse languages were selected, covering families such as Germanic, Slavic, Indo-Iranian, Uralic, Semitic, Dravidian, Austronesian, and Niger-Congo.
Baselines: Native English models were trained on the original FineWeb corpus with matched data sizes to serve as a performance upper bound.

Evaluation Metrics

The models were evaluated using three primary methods:

Perplexity (PPL): Measured on native English text from FLORES+ (Wikinews, Wikitravel, Wikibooks) and a subset of FineWeb. This assesses general language modeling capability.
Linguistic Acceptability: Evaluated using BLiMP (Benchmark of Linguistic Minimal Pairs). This tests the model's ability to distinguish grammatically correct sentences from incorrect ones across 12 categories (e.g., agreement, island effects, filler-gap dependencies).
Cross-Evaluation Perplexity: Models trained on translations from Language A were tested on translations from Language B. This measures the similarity of "translationese dialects" between different source languages.

Analysis Variables

The authors correlated performance with three predictors:

Base Corpus Size: The size of the source language corpus before translation.
Lexical Diversity: Measured by Type-Token Ratio (TTR) and Bigram TTR in the translated corpus.
Syntactic Similarity: Calculated using lang2vec based on WALS (World Atlas of Language Structures) features to determine the typological distance between the source language and English.

3. Key Contributions

Systematic Analysis of Source Language Impact: The paper provides the first large-scale, controlled study analyzing how 24 diverse source languages affect the learning of English language models when the data is machine-translated.
Differentiation of Learning Goals: It distinguishes between general language modeling (perplexity) and grammatical competence (syntactic acceptability), showing they are driven by different factors.
Identification of "Translationese Dialects": The study demonstrates that models trained on different source languages develop distinct "dialects" of translationese, where typologically similar source languages produce mutually more intelligible translated corpora.
Data Scale Effects: It highlights that the importance of typological similarity increases significantly as the data scale grows from 100MB to 1000MB.

4. Key Results

A. General Language Modeling (Perplexity)

Translationese Penalty: No model trained on translated data matched the native English baseline. The gap was particularly large on heterogeneous web text (FineWeb) compared to structured text (Wikibooks).
Lexical Diversity vs. Data Scale:
- Low Data (100MB): General perplexity was strongly driven by the lexical diversity (TTR) of the translated corpus. Higher diversity correlated with lower perplexity.
- High Data (1000MB): Lexical diversity ceased to be a significant predictor.
Typological Similarity: At the 1000MB scale, syntactic similarity to English became a significant predictor of lower perplexity, though some typologically distant languages (e.g., Maltese, Zulu) still performed competitively, suggesting domain coverage also matters.

B. Grammatical Competence (BLiMP)

The Dominance of Typology: Unlike perplexity, grammatical performance was not driven by lexical diversity or corpus size.
Strong Correlation: At the 1000MB scale, there was a strong, significant positive correlation between syntactic similarity to English and BLiMP accuracy ( $r \approx 0.61$ ).
Specific Phenomena: This correlation was strongest for complex syntactic phenomena, particularly agreement (Anaphor, Subject-Verb) and long-distance dependencies (Island Effects, Filler-Gap).
Low Data Insensitivity: At 100MB, typological similarity did not significantly predict grammatical accuracy, suggesting that small models cannot yet leverage structural similarities effectively without sufficient data.

C. Cross-Evaluation Perplexity

Mutual Intelligibility of Dialects: Models trained on translations from Language A performed significantly better (lower perplexity) on translations from Language B if A and B were typologically similar.
Implication: This confirms that different source languages generate distinct "dialects" of translationese. Typologically similar languages result in translated English that shares similar structural artifacts.

5. Significance and Implications

Strategic Data Selection: For building multilingual models, the choice of source language is critical.
- If the goal is general fluency/perplexity in low-data regimes, prioritizing source languages with high lexical diversity in their native corpora is beneficial.
- If the goal is grammatical competence (especially for complex syntax), prioritizing typologically similar source languages is essential, provided sufficient data volume is available.
The "Impoverishing" Effect of Translation: The study confirms that training on translated data generally leads to a flattening of linguistic capabilities compared to native data, particularly in grammatical acceptability.
Upper Bound Estimation: Since the study used translations into English (a high-resource language where MT is strong), the results likely represent an upper bound for translated data quality. The negative effects of translationese may be even more severe when translating into low-resource languages.
Future Directions: The findings suggest that simply scaling data is insufficient; the structural quality and typological alignment of the data become the limiting factors for grammatical learning in large-scale settings. Future work should explore disentangling structural similarity from lexical overlap and testing these patterns on larger foundation models.

Conclusion

The paper concludes that the source language of machine-translated training data shapes learned linguistic knowledge in systematic ways. While lexical diversity drives general language modeling in low-data settings, typological similarity becomes the dominant factor for grammatical competence as data scales increase. This implies that for high-quality grammatical learning, the structural proximity of the source language to the target language is a more critical design choice than the sheer volume or diversity of the raw source text.