A Multilingual Dataset of Lexical, Syntactic, and Semantic Features from Wikipedia across Five Languages
This paper introduces a publicly available multilingual dataset comprising lexical, syntactic, and semantic features from 1,860 Wikipedia articles across five languages, demonstrating through statistical validation that language significantly influences these features independent of text length, thereby offering a robust resource for crosslinguistic and computational linguistic research.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Language is often described as a collection of rules, but to a computer, it is a landscape of patterns. For decades, linguists have tried to map these patterns by hand, cataloging how different tongues structure their sentences or how varied their vocabulary is. These manual maps are valuable, but they are slow to make and often miss the subtle, continuous flow of how people actually write. In recent years, powerful computer programs have emerged that can read text and translate it into mathematical shapes, capturing the "feel" of a sentence in a way that humans cannot easily see. This new approach allows researchers to compare languages not just by their dictionary definitions, but by the deep, statistical fingerprints left behind in millions of words. The question remains: do these digital fingerprints reveal that languages are fundamentally different, or are they just variations on a similar theme?
A researcher at the Queensland University of Technology has taken a significant step toward answering this by building a massive, shared library of text from five major languages: English, Chinese, Spanish, French, and German. Instead of relying on a few famous examples, the team gathered nearly 1,900 encyclopedia articles, carefully selecting topics that appear in all five languages, such as history, science, and politics. To ensure a fair comparison, every article was trimmed to the same length, creating a level playing field where the only real difference was the language itself. The team then ran each article through a sophisticated analysis pipeline. They counted how many unique words appeared, measured how often words repeated, and tracked how far apart words were from the words they depended on in a sentence. They also used a modern artificial intelligence tool to convert the meaning of the first twenty sentences of each article into a long list of numbers, creating a unique digital signature for the text's semantic content.
The results of this comparison were striking. When the researchers plotted all these features together, the language itself proved to be the strongest force shaping the data. It explained more than a third of the differences found across the entire collection of articles. In contrast, the length of the text or the specific method used to find the articles had almost no effect on the overall pattern. This confirmed that the distinct "voice" of each language is a robust feature that persists even when the topic and length are held constant. The analysis showed that Chinese stood apart from the others, forming a distinct group, while the European languages clustered more closely together. English and Spanish were found to be the most similar pair, sharing the closest statistical profile.
The study also uncovered a deep connection between how a language is structured and what it means. The researchers found a clear link between the distance of words in a sentence and the distance of their meanings in the digital space. This suggests that the way a language arranges its words is not random but is tightly woven with the concepts it expresses. While the dataset is currently limited to these five languages and relies on encyclopedia writing, it provides a solid, reproducible foundation for future work. It offers a new way to test how well computer programs understand different tongues and helps scientists predict how languages might evolve or relate to one another. By turning the complexity of human speech into a clear, measurable map, this work allows us to see the hidden architecture of language with a precision that was previously impossible.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.