Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders

Large language models are the engines behind many of today's most advanced artificial intelligence tools, capable of reading, writing, and solving problems in dozens of languages. For years, researchers have wondered how these systems handle the complexity of being multilingual. When a model solves a math problem in English and then solves the exact same problem in Chinese, does it use the same internal mental machinery for both, or does it switch to a completely different set of rules for each language? This question matters because if the model uses a shared internal language for reasoning, we could understand its logic once and apply that understanding to every language it speaks. If, however, it relies on separate, language-specific pathways, then understanding one version of the model tells us very little about the others. The field of mechanistic interpretability attempts to look inside these models to see how they work, treating their internal states like a complex circuit board that can be mapped and understood.

A team of researchers set out to answer this question by peering into the internal workings of five different large language models. They focused on a specific task: solving elementary school math problems. They took a dataset of 250 math problems and asked each model to solve them in six different languages: English, German, French, Spanish, Russian, and Chinese. To ensure they were comparing apples to apples, they only kept the problems where the model got the right answer and produced a clear, step-by-step reasoning path in every single language. The researchers then recorded the model's internal activity at various stages of the process, capturing the specific patterns of electricity-like signals that fired inside the computer as it thought through the solution.

To make sense of these complex signals, the researchers used a tool called a sparse autoencoder. You can think of this as a way to translate the model's chaotic internal noise into a list of distinct, understandable concepts. Just as a human might describe a scene by listing its key elements—sky, tree, dog, car—the autoencoder breaks down the model's activity into a set of active features. The researchers trained two versions of this tool for each layer of the models. The first version simply learned to describe what it saw in one language. The second version was trained with a special instruction: it had to learn to describe the same math problem using the same set of features, regardless of whether the problem was presented in English or Chinese. This second version was designed to force the model to find a common ground, a shared internal language for reasoning.

The results revealed a surprising reality about how these models think. The researchers found that simply making the internal descriptions look more similar geometrically did not guarantee that the model was actually using the same logic. In some models, forcing the internal features to align across languages made the model's reasoning more robust and interchangeable. In others, it made no difference at all, and in one case, it actually made the model's performance worse when they tried to swap parts of the reasoning from one language to another. This happened because the models had already settled into very different habits. One family of models naturally developed a shared way of thinking as it got deeper into the problem, so the new training helped refine that shared path. Another family had already built such a strong, shared structure that the new training had nothing to add. A third family kept its languages strictly separate, and the attempt to force them together created confusion rather than clarity.

The study examined five specific models from four different families, ranging in size from 1.7 billion to 4 billion parameters. They discovered that the success of creating a shared reasoning language depended entirely on the model's existing architecture. In the models that showed a "convergent" pattern, where the shared features grew stronger as the model processed more of the problem, the new training method worked perfectly, improving the model's ability to reason across languages in every single layer they tested. In models that were already "saturated" with shared features, the new training provided no benefit and sometimes disrupted the model's natural flow. In models that remained "low-sharing," where the languages stayed mostly separate, the new training failed to create a functional bridge, even though it made the internal signals look more similar on paper.

This work challenges a common assumption in the field: that if two things look similar on a graph, they must function the same way. The researchers proved that a model can have features that look geometrically aligned across languages but still fail to work interchangeably. They showed that the path to understanding multilingual models is not a one-size-fits-all solution. Instead, the internal logic of a model is deeply tied to its specific design. Some models naturally build a universal reasoning engine, while others keep their languages in separate rooms. The researchers concluded that to truly understand how an artificial intelligence thinks across languages, we must look at the specific architecture of each model, rather than assuming they all follow the same hidden rules.