Automatically Detecting Code-Switching Between Nigerian Pidgin English and English

In the bustling, multilingual landscapes of Nigeria, language is rarely a single, static stream. Instead, it is a dynamic current where two tongues often flow together in the same sentence. This practice, known as code-switching, happens when a speaker moves fluidly between Standard English and Nigerian Pidgin English, a vibrant creole language used by millions as a bridge across diverse ethnic groups. For computers, which rely on rigid rules and clear categories to understand text, this fluidity presents a significant puzzle. Traditional tools often struggle to tell where one language ends and the other begins, especially when the second language, Pidgin, lacks a single standardized spelling and varies widely by region. Without a way to accurately identify these switches, computers cannot effectively process the rich, mixed-language conversations that define daily life for millions of people, leaving a vast portion of human communication invisible to digital technology.

To solve this problem, researchers at Bangor University set out to build a new kind of map for these mixed-language texts. They began by gathering a massive collection of real-world writing, pulling together stories from thirty different books, thousands of comments from an online forum, and hundreds of news articles. This diverse mix ensured they captured the full range of how people actually speak and write, from the formal structure of news reports to the casual, rapid-fire exchanges of internet users. The team then created a "ground truth" dataset, a carefully verified reference guide where human experts manually marked every single point where the language switched from English to Pidgin or back again. This human-verified map served as the gold standard against which all computer models would be tested.

The researchers then trained several different types of computer programs to learn how to spot these switches on their own. They tried the usual suspects in the field of artificial intelligence, such as deep learning models that attempt to mimic the way human brains process sequences of words. They also tested traditional statistical classifiers that look for patterns in groups of letters. However, the most surprising and effective tool turned out to be something quite different: a compression-based model. In simple terms, this type of program works by learning the statistical patterns of how letters follow one another in a specific language, much like a person who has read a book so many times they can predict the next word without thinking. The researchers used a specific method called Prediction by Partial Matching, which allows the computer to analyze text character by character, building a probability map of what comes next based on the immediate context.

When the team put these different programs to the test against their human-verified map, the results were revealing. The deep learning models, which often require enormous amounts of data to work well, struggled to keep up, achieving lower accuracy scores than expected. The traditional statistical models performed better, with one specific type of random forest classifier reaching an accuracy of 89 percent. Yet, the compression-based models, which do not need massive datasets to learn, performed just as well as the best traditional models, also hitting an accuracy of 89 percent. What made these compression models particularly impressive was their speed and efficiency. While other models took longer to train and process, the compression models could learn the patterns of the language and then analyze new text in just a few seconds, requiring very little computing power.

The study also uncovered interesting details about the languages themselves. By analyzing the text, the researchers found that Nigerian Pidgin tends to use a more compressed vocabulary, relying heavily on a smaller set of high-frequency words compared to the broader, more diverse vocabulary of Standard English. This difference in structure helped the computer models distinguish between the two, as the statistical patterns of letter sequences in Pidgin were distinct enough to be recognized even without a massive dictionary. The confusion matrices, which show exactly where the models made mistakes, revealed that the compression models were particularly good at identifying Pidgin, correctly flagging nearly 93 percent of the Pidgin segments, though they occasionally mistook some English for Pidgin.

Ultimately, this research demonstrates that you do not always need the most complex, data-hungry artificial intelligence to solve difficult language problems. For low-resource languages like Nigerian Pidgin, which lack the vast digital archives available for major world languages, simpler, statistical approaches can be just as effective as the most advanced neural networks. The compression-based models proved to be a powerful, fast, and efficient alternative, capable of segmenting mixed-language text with high precision. By providing a new, manually verified dataset and proving the viability of these lightweight models, the researchers have opened a door for future tools that can finally understand and process the rich, code-switching conversations of millions of people, making digital technology more inclusive for speakers of diverse and evolving languages.