Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
This paper demonstrates that word order biases in decoder-only language models are not inherent architectural preferences but are instead driven by training data, with models favoring right-branching SVO structures in natural languages due to data abundance and quality, thereby posing a risk of reducing global word order diversity.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Language is not just a collection of words; it is a set of rules for how those words are arranged to make sense. In some languages, the verb comes after the subject and object, like "The cat ate the fish." In others, the verb comes first, or the object comes before the subject. This arrangement, known as word order, is one of the most fundamental ways human languages differ. For decades, linguists have studied these patterns to understand how the human brain learns and processes language, looking for universal rules that might apply to all people everywhere. Today, a new kind of learner has entered the field: artificial intelligence. These computer programs, called language models, are trained on vast amounts of text and can generate human-like sentences. A critical question has emerged: do these machines learn language the way humans do, or do they develop their own hidden preferences for how words should be ordered?
A team of researchers set out to answer this by testing how these models handle different word orders. They built a controlled experiment using two very different types of data. First, they created hundreds of artificial languages. These were not real languages spoken by people, but mathematical constructs designed to isolate specific features. Imagine a language where the only thing that changes is whether the adjective comes before or after the noun, or whether the verb comes at the beginning or end of a sentence. By stripping away the complexity of real human speech—such as cultural context, irregular grammar, and messy data—the researchers could see if the computer model had a built-in preference for one structure over another. They then compared these results with how the same models performed on real, natural languages from around the world, ranging from English and Spanish to Japanese and Swahili.
The findings revealed a surprising split in how these models behave. When the researchers trained the models on the artificial languages, the computers consistently preferred a specific structure where modifiers came before the main word, a pattern linguists call "left-branching." This preference was so strong that it appeared regardless of the base order of the sentence. However, this preference did not match what we know about human language learning. Humans tend to prefer consistency; if a language puts the verb at the end, humans usually expect other parts of the sentence to follow a similar pattern. The artificial languages showed that the models were not following human logic or the statistical patterns found in the world's languages. Instead, the models seemed to favor a structure that was simply easier for their specific architecture to process, a bias that existed independently of any real-world language rules.
The story changed completely when the researchers switched to real-world data. When the models were trained on natural languages, but with only a small amount of text—about five megabytes per language—they showed no clear preference for any specific word order. They were equally comfortable with different arrangements. However, as the researchers increased the amount of training data to a much larger scale, around one thousand megabytes, a distinct pattern emerged. The models began to favor languages where the subject comes before the verb, and the verb comes before the object, such as English or Chinese. This is known as the Subject-Verb-Object, or SVO, order. Interestingly, this preference grew stronger as the models saw more data, even though another order, where the verb comes at the end (Subject-Object-Verb), is actually more common across the world's languages.
The researchers investigated why this shift happened. They found that the preference for SVO was not because the computer architecture was inherently better at understanding that specific order. Instead, the preference was driven entirely by the data the models were fed. The languages that are most widely available on the internet and in digital libraries—often called "high-resource" languages—are overwhelmingly SVO. Because the models were trained on massive amounts of text from these specific languages, they learned to prioritize that structure. When the researchers looked at models trained on languages with very little available data, the SVO advantage disappeared. This suggests that the models are not discovering a universal truth about language; they are simply reflecting the imbalance in the information they have access to.
This discovery has significant implications for the future of language technology. If these models are increasingly used to translate, write, or communicate in languages that are less common or have different word orders, there is a risk that they will subtly push those languages toward the SVO structure. The models might make it harder to use or understand the natural, flexible word orders that many languages rely on for expressing nuance, focus, or emotion. The study shows that while the computer's internal design might have a slight bias, the data it consumes is the dominant force shaping its behavior. As these tools become more widespread, the way we curate and select the data they learn from will determine whether they preserve the rich diversity of human language or gradually smooth it out into a single, uniform pattern.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.