Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs
This paper introduces J-PragEval-v0, a minimal-pair benchmark for Japanese pragmatic phenomena, and uses linear probing and activation steering on a 1.5B bilingual model to demonstrate that while honorifics are linearly decodable in residual streams, other pragmatic contrasts like implicit subjects and in-group reference are resolved during generation rather than stored at the prompt.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Language is more than a collection of words and grammar rules; it is a living system of social cues, unspoken agreements, and cultural context. When a person speaks Japanese, they are constantly navigating a complex map of relationships, deciding who is inside their circle of trust and who is outside, and choosing words that reflect exactly how they feel about the person they are addressing. This is the realm of pragmatics, the study of how context shapes meaning. For decades, artificial intelligence has excelled at translating words and answering questions, but it often stumbles when asked to understand these subtle social layers. A computer might know the dictionary definition of a polite phrase, but it frequently fails to know when to use it, or worse, uses it in a way that sounds rude or strange to a human listener. This gap is particularly wide for Japanese, a language where the relationship between speakers dictates the very structure of the sentence. As researchers build smaller, more efficient computer models designed to run on everyday devices, a critical question has emerged: do these compact models actually understand the cultural rules of the languages they speak, or are they merely mimicking the surface patterns?
A recent study by independent researcher Florian Braun tackles this question by looking inside a small, bilingual computer model that speaks both English and Japanese. The research focuses on a specific type of artificial intelligence known as a small language model, which is designed to be powerful enough for useful tasks but light enough to run without massive, energy-hungry servers. While these models are becoming common in Japan, they are often tested only on how well they translate text or answer reading comprehension questions. These standard tests, however, miss the most difficult part of human communication: the ability to read the room. To fix this, the researcher created a new testing ground called J-PragEval. Instead of asking the model to translate a sentence, this test presents the model with two nearly identical scenarios that differ only in a single social detail, such as whether the speaker is talking to a boss or a friend. The model must then choose the correct way to finish the sentence based on that social detail. The test covers four specific areas of Japanese social life: the use of honorifics to show respect, the ability to guess who is being talked about when the subject is dropped from the sentence, the choice of verbs that show whether someone is an insider or an outsider, and the art of refusing a request politely without saying a direct "no."
The researcher then peered inside the computer model, which has 28 layers of processing, to see where it stores the knowledge needed to pass these tests. Using a method that acts like a microscope for the model's internal thoughts, the study looked at the model's activity at different stages of processing. The results revealed a surprising split in how the model handles these social rules. For the use of honorifics, the model behaves like a well-organized library. The information about which level of politeness to use is clearly stored in a specific layer of the model, and a simple check can find it with nearly 96 percent accuracy. The model knows the rule, and it keeps that rule ready to use. However, the picture changes for the other social rules. When the model had to decide who was being talked about in a sentence with a missing subject, or choose the right verb for an insider versus an outsider, the researchers could not find a clear, static memory of these rules inside the model at the moment it finished reading the prompt. The internal signals were too faint to be detected by the standard check. Yet, when the model actually generated an answer, it got the right choice about 77 to 79 percent of the time. This suggests that the model does not simply retrieve a stored rule for these tasks; instead, it figures out the answer on the fly, computing the social context as it builds the sentence word by word.
The study also uncovered a trap in how we test these models. One of the four tests, which focused on indirect refusals, initially seemed to show that the model had mastered the skill, with a detection score of 95 percent. However, when the researcher looked closer at how the model actually behaved, it turned out the model was failing this test more often than not. The high score was a false alarm caused by a flaw in the test design itself: the correct answers happened to be longer sentences, and the model was simply guessing based on sentence length rather than understanding the social nuance. Once the test was adjusted to account for length, the model's true inability to handle indirect refusals was revealed. This finding serves as a warning that high scores on computer tests can sometimes hide deep misunderstandings if the tests are not carefully designed to separate true understanding from surface tricks.
To address these findings, the researcher proposed a new way to guide these models without needing to retrain them from scratch. The idea, called Pragmatic Representation Steering, involves gently nudging the model's internal activity in the right direction at the moment it is generating text. Think of it as a subtle adjustment to the model's internal compass rather than a complete overhaul of its brain. The study showed that the mathematical direction needed to make these nudges exists within the model's structure, at least for the rules that are clearly stored. This suggests that it is possible to make small language models more culturally aware and socially appropriate without the massive cost of full retraining. While the full test of this steering method on larger models is planned for future work, the current study proves that the internal machinery for cultural understanding is present, even if it is hidden in different ways for different social rules. The research concludes that for small language models to truly serve Japanese users, we must look beyond simple translation scores and understand how these models process the invisible, unspoken rules of human connection.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.