Wiktionary as a Crowdsourced Lexicon for English Dialects

Language is not a single, solid block; it is a living landscape that shifts and changes depending on where you stand. A word that means one thing in London might mean something entirely different in Auckland, or might not exist at all in Mumbai. For decades, the people who map these linguistic territories have relied on traditional dictionaries, books compiled by teams of experts who spend years deciding which words are important enough to include. But the world of language is moving faster than any team of editors can keep up with, especially for the many varieties of English spoken around the globe. As computers and artificial intelligence become more common, there is a growing need to teach them how to understand these local differences. If a computer cannot tell the difference between a New Zealand phrase and an American one, it will fail to understand the people using it. This creates a problem for fairness in technology, as systems built only on standard English often struggle with the rich, diverse ways people actually speak.

To solve this, researchers are looking for new ways to gather language data that is both vast and up-to-date. One promising source is Wiktionary, a free, online dictionary that anyone can edit. Unlike traditional books, which are written by a small group of paid experts, Wiktionary is a crowdsourced project where volunteers from around the world add entries, definitions, and examples. The question researchers asked was simple but difficult: Can this open, community-built dictionary serve as a reliable map for the many different dialects of English? Specifically, they wanted to know if Wiktionary covers regional words as well as the famous Oxford English Dictionary, and if the words found there actually appear in the real conversations people have on social media.

The researchers began by comparing the contents of Wiktionary against the Oxford English Dictionary for twelve different national varieties of English, ranging from the well-known British and American English to varieties from India, Kenya, and the Philippines. They found that while the traditional dictionary still holds a slight edge in size for standard American and British English, Wiktionary actually covers more ground for many other regions. For varieties like Canadian, Irish, and Australian English, Wiktionary contained significantly more entries than the traditional dictionary. The difference was even more striking for English spoken in countries outside the traditional "inner circle" of English-speaking nations. While the traditional dictionary had no entries at all for Kenyan or Pakistani English, Wiktionary had hundreds of entries for these varieties. This suggests that the community-driven model is remarkably good at capturing words that professional editors might overlook or not yet have time to include.

To test if these words were actually being used in the real world, the team turned to social media. They looked at millions of posts and comments from country-specific communities on Reddit, a popular online forum. They treated these digital communities as a proxy for geographic location, assuming that people posting in a New Zealand community were likely using New Zealand English. They then checked to see if the words listed in Wiktionary for each country appeared frequently in the corresponding online community. The results showed a strong connection. When looking at the main posts people wrote, the words found in Wiktionary appeared exactly where they were expected to be. For example, words specific to New Zealand showed up most often in the New Zealand community, and words for India appeared most often in the Indian community. This alignment suggests that Wiktionary is not just a list of words, but a reflection of how people actually speak in their daily digital lives.

However, the study also revealed that the type of text matters greatly. The connection between the dictionary and the social media data was very strong when looking at the main posts people wrote, but it became much weaker when looking at the comments people left in response. In the comment sections, the language was more conversational and often mixed with words from other languages. For instance, in the New Zealand community, some words that looked like New Zealand English actually turned out to be common words in Tagalog, the language of the Philippines, because the two languages share some similar spellings. This highlighted a challenge: while the dictionary is useful, the way people use language in fast-paced, interactive conversations can be messy and influenced by many different factors. The researchers found that the dictionary worked best when the text was more formal or structured, like a main post, rather than in the quick back-and-forth of a comment thread.

The study also looked closely at how words are formed. They compared the types of new words found in Wiktionary for New Zealand English against the traditional dictionary. They found a very high level of agreement in the patterns used to create these words, such as combining two words together or changing a word's meaning slightly. This suggests that even though Wiktionary is written by volunteers, the way they build entries follows the same logical rules as professional linguists. The researchers noted that Wiktionary was particularly good at including phrases and idioms—groups of words that have a specific meaning together—which are often missing from traditional dictionaries. This makes the crowdsourced resource especially valuable for understanding the colorful, expressive side of language that formal books sometimes leave out.

Ultimately, the paper concludes that Wiktionary is a powerful and viable tool for understanding English dialects, especially for regions that have been historically underrepresented in language technology. It does not replace the need for professional dictionaries, but it complements them by offering a faster, more inclusive, and more up-to-date view of how language is evolving. The researchers caution that using this data requires care, particularly when dealing with online conversations where different languages might mix or where the context of the conversation changes the meaning of a word. But for the goal of building fair and accurate language tools that work for everyone, not just the majority, this open, community-built dictionary offers a crucial piece of the puzzle. It shows that when people come together to document their own language, they create a resource that is both deeply accurate and remarkably broad.