Non Monotonic Cross Lingual Representational Distances for Culturally Specific Japanese Concepts in Two Transformer Models

Inside the digital minds of modern artificial intelligence, there exists a vast, invisible landscape where words and ideas are stored not as definitions, but as points in space. Researchers call these points "representations." When a computer reads a sentence, it translates the text into a long list of numbers that describe the meaning of that sentence relative to everything else it knows. If two ideas are very similar, their points in this space sit close together; if they are different, they sit far apart. For years, scientists have known that these machines can move information between languages, translating a thought from English to Japanese or Spanish without losing its core shape. But a lingering question remains: does the machine treat a culturally unique idea the same way it treats a standard one? Does the specific flavor of a culture, with its unique social rules and values, leave a distinct mark on how the machine "sees" the words, or does the machine simply smooth those differences away?

This question drove a recent study that looked inside two popular artificial intelligence models to see how they handle twenty specific Japanese concepts that are deeply rooted in their culture. These are not just ordinary words; they are terms like giri, which describes a deep sense of social duty, or wa, which refers to the harmony of a group. The researchers wanted to know if the machine's internal map of these ideas looked different when the concept was presented in its native Japanese compared to when it was described in English or Spanish. They were not asking if the machine could translate the words correctly, but rather how the distance between the ideas changed as the information traveled through the different layers of the computer's brain.

To find the answer, the researchers set up a careful experiment using two different AI models, one slightly larger than the other. They took twenty of these culturally specific Japanese concepts and created four different versions of a prompt for each one. The first was the original Japanese sentence. The second was a direct, word-for-word English translation of that Japanese sentence. The third was a Spanish translation of the standard English baseline. The fourth was a scrambled version of the Japanese text, where the order of the computer's internal code was shuffled to break the meaning while keeping the same letters and sounds. By feeding these four versions into the models, the team could measure how far apart the resulting internal points were from a standard English baseline. They measured this distance at every single step, or "layer," of the model's processing, from the very first moment the text was read to the final moment before an answer was generated.

The results revealed a pattern that was surprisingly consistent across the board, regardless of the language or the specific concept. As the information moved through the layers of the models, the distance between the ideas did not simply get smaller or larger in a straight line. Instead, it followed a curve. The distance started high, dropped down to a low point very early in the process, and then began to rise again as the information moved toward the end. This "dip and rise" happened for the Japanese prompts, the Spanish prompts, the literal English translations, and even the scrambled Japanese text. The fact that the scrambled text, which had no real meaning, showed the same pattern suggests that this behavior is a fundamental property of how these models process any sequence of tokens, rather than a special signature of cultural depth.

However, when the researchers focused on a specific middle section of the model's layers, a clear difference emerged. In this range, the Japanese prompts containing the culturally specific concepts were consistently farther away from the English baseline than the other versions were. Whether comparing Japanese to literal English or Japanese to Spanish, the Japanese ideas occupied a more distant position in the machine's internal space. The difference was measurable and significant, with the Japanese concepts sitting noticeably further out than their counterparts. This held true for all twenty concepts in both models tested. The data showed that the machine did not treat these culturally rich ideas as identical to their English descriptions; it kept them at a distinct distance, suggesting that the cultural framing or the specific linguistic structure of the Japanese input left a lasting mark on the representation.

Despite this clear distance, the study stops short of declaring that the machine fails to understand the culture or that it cannot translate the meaning. The researchers were careful to note that the Japanese prompts differed from the English ones not just in culture, but in language, sentence structure, and the specific motivations described in the text. Because so many factors changed at once, it is impossible to say for certain that the extra distance was caused solely by the cultural uniqueness of the words. It could just as easily be the result of the language itself or the way the sentences were constructed. Furthermore, the fact that the scrambled text also showed the early dip and later rise means that the non-monotonic pattern is not a unique fingerprint of cultural meaning.

The study concludes that while these models do show layer-dependent differences in how they represent culturally specific Japanese concepts, the reasons for those differences remain a mix of language, structure, and culture that cannot be fully separated with this experiment alone. The machine does not simply flatten all cultural nuances into a single point; it maintains a measurable distance for these specific ideas. Yet, this distance does not prove that the machine has lost the meaning or that it is struggling with the culture. It simply shows that the internal geometry of these models is complex and sensitive to the specific way a concept is framed. The findings offer a precise map of where these ideas sit in the machine's mind, but they leave the deeper question of why they sit there, and what that distance truly means for human understanding, as an open invitation for future research.