The Geometry of Low-Resource Language Representations
This paper demonstrates that low-resource languages in large language models suffer from representational degeneration in final layers due to data scarcity, and shows that applying geometric regularisation during continued pretraining can effectively mitigate this issue to improve performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Large language models are the engines behind many of the artificial intelligence tools we use today, capable of writing stories, solving problems, and translating text. However, these systems are not built equally for every human language. While they perform brilliantly with languages like English or Spanish, which have vast amounts of digital text available for training, they often struggle significantly with languages that have fewer digital resources. This gap is not just a matter of having less data; it suggests that the internal way these models process information breaks down when the data is scarce. Scientists have long wondered what happens inside the model's "brain" when it encounters a language it hasn't seen enough of, and whether the way it organizes meaning changes depending on how much information it has to work with.
A team of researchers from the University of Cape Town set out to look directly inside these models to find the answer. They focused on the geometry of the model's internal representations. In simple terms, when a computer processes a word, it turns it into a list of numbers that represents its meaning. These lists of numbers exist in a vast, multi-dimensional space. The researchers wanted to see how the shape of this space changes for different languages. They discovered that for languages with abundant data, these numerical lists are spread out and distinct, allowing the model to keep different meanings separate. But for low-resource languages, the model seems to collapse this space, forcing many different meanings into a tight, crowded cluster where they lose their distinctiveness. This phenomenon, which the researchers call representational degeneration, means the model is effectively running out of room to think clearly about these languages.
To investigate this, the team examined nine different large language models, ranging from smaller 1-billion-parameter versions to larger 12-billion-parameter ones. They analyzed how these models represented thirty different languages, from very high-resource languages like English to very low-resource languages like Oromo and Wolof. By measuring how close the numerical representations of words were to each other, they found a clear pattern: the less data a language had, the more the model's internal representations collapsed together. This effect was most pronounced in the final layers of the models, which are the parts responsible for making the final decision about what a word means or what comes next. In these final stages, the model seemed to lose its ability to distinguish between different concepts in low-resource languages, creating a bottleneck that limits performance.
The researchers then asked if they could fix this problem. They knew that simply adding more training data was often impossible for these languages, so they looked for a way to guide the model's internal geometry during a process called continued pretraining. This is a method where a model that has already been trained on a broad mix of languages is given a second, focused round of training on a specific low-resource language. The team introduced a new technique that acted as a gentle guide during this training. They added a rule that penalized the model if it let the numerical representations of words get too close to one another. This rule forced the model to keep the representations spread out and distinct, effectively preventing the collapse they had observed.
They tested this approach by adapting nine different base models to ten African languages, including Kinyarwanda, Hausa, and Yorùbá. The results showed that this geometric guidance worked. The models trained with this new rule maintained a healthier, more spread-out internal structure for these languages compared to models trained without it. When they tested the models on difficult tasks like answering questions or solving math problems, the improvements were most noticeable in the larger models. For these bigger systems, the geometrically regularized training led to better performance, particularly on the most challenging tasks. The study suggests that the key to helping these models understand low-resource languages better is not just feeding them more text, but ensuring that the way they organize that text remains clear and distinct.
This work highlights a fundamental truth about how these artificial minds learn: the amount of data available shapes the very structure of their thinking. When data is scarce, the internal space where meaning lives becomes cramped and inefficient. By recognizing this geometric flaw, the researchers demonstrated that it is possible to intervene and correct it. While the improvements were modest in some areas, the fact that a simple adjustment to the training process could restore the model's ability to distinguish between concepts offers a promising path forward. It suggests that with the right kind of guidance, we can help these powerful tools serve all human languages more effectively, bridging the gap between the rich and the resource-poor without needing to wait for more data to appear.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.