A Study of Crosslinguistic Influence in Language Models
This study systematically investigates crosslinguistic influence in language models, revealing that structural interference is a structured phenomenon governed by the interplay of language dominance and proficiency, where increased L1 dominance amplifies syntactic distance effects while limited L2 proficiency hinders positive transfer, with deep-layer attention mechanisms physically routing L1 priors to drive these cross-lingual interactions.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot how to speak. You start by feeding it a massive library of stories in one language, say, German, Spanish, or Japanese. The robot gets really good at it, learning the rhythm, the grammar, and the flow. But then, you decide to teach it a second language, English. What happens inside the robot's "brain" when it tries to learn this new language after it has already mastered the first one? This is the world of Crosslinguistic Influence (CLI). In human psychology, we know that if you learn a second language after your first, your first language doesn't just sit quietly; it actively interferes or helps. If the two languages are similar, like Spanish and Italian, your brain uses the old rules to speed things up. If they are very different, like English and Japanese, the old rules can get in the way, causing you to make mistakes. Scientists have long wondered if artificial intelligence, specifically Large Language Models (LLMs), experiences this same "bilingual brain" struggle, or if it's just a random glitch in the code. Understanding this is crucial because if we want AI to speak many languages fluently without mixing them up, we need to know exactly how these languages interact inside the machine.
In this study, researchers at Maastricht University decided to play the role of a strict language teacher for a small robot brain (a GPT-2 model). They wanted to see what happens when they change the order and timing of how the robot learns. They created a scenario where the robot learned a first language (L1) for a while, and then, at different points in its training, they introduced a second language (L2). They called this timing the "Step of Exposure." Think of it like this: if you start learning French immediately after English, you might mix them up easily. But if you spend a whole year mastering your first language before even seeing an English word, your first language becomes so strong (or "dominant") that it might be harder to learn English later without your first language brain getting in the way.
The team tested this with 15 different first languages, ranging from those very similar to English (like German) to those very different (like Japanese). They didn't just ask the robot to chat; they used a clever trick called structural priming. Imagine showing the robot a sentence in its first language, and then immediately asking it to complete a sentence in the second language. It's like whispering a hint in the robot's ear before it answers. They found that this hint had a massive effect. If the first language was similar to the second, the hint helped the robot get the grammar right (positive transfer). But if the languages were far apart, the hint actually made the robot more likely to make mistakes (negative transfer).
Here is where it gets really interesting: the researchers discovered a tricky trade-off. The longer they let the robot practice the first language before introducing the second (a higher "Step of Exposure"), the more the robot's first language dominated its thinking. This dominance made the robot very sensitive to how similar the two languages were. However, a critical problem emerged: as the first language became too strong, the robot actually lost its ability to use the helpful hints for similar languages (positive transfer degraded). Yet, the harmful interference from very different languages (negative transfer) didn't go away; it stubbornly persisted and even got worse. The study suggests that when the first language is too strong, the robot loses its capacity to learn the new language properly, and the old rules start to force their way into the new sentences, even when they don't belong.
To understand how this happens inside the robot's "mind," the researchers looked at the specific parts of the code that were lighting up. They found that when the robot was just learning normally, the parts of the brain that handle grammar were active in the middle layers of the network. But when they gave it that "whispered hint" (the prime), the robot's brain shifted gears. The grammar processing physically moved to the very last layers of the network, right before the robot gave its final answer. It was as if the robot had to wait until the very end to decide how to mix the two languages.
Furthermore, they found that the "attention" mechanism—the part of the AI that decides which words to focus on—was the real boss of this interaction. When the robot tried to use a hint from a similar language, the attention heads helped it connect the dots. But when the hint came from a very different language, those same attention heads forced the wrong rules onto the new sentence. The study rules out the idea that this is just a random error or a result of bad translation; instead, it shows that this is a structured, predictable phenomenon. The robot isn't just confused; it's actively trying to use its old knowledge, and sometimes that old knowledge is exactly what it needs, and sometimes it's exactly what it doesn't.
Ultimately, the paper suggests that Crosslinguistic Influence in AI isn't a bug; it's a feature of how these models learn. It mirrors how humans learn, where the strength of your first language and how well you know the second one dictate whether you get a helpful boost or a confusing interference. The researchers conclude that to build better multilingual AI, we can't just throw all languages into the mix at once. We have to be careful about how much time we spend on the first language before introducing the next, because if the first language gets too strong, it might permanently block the robot from learning the new one correctly.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.