Beyond the pale: Assessing prevalence and contents of extremist speech in LLM training data

Large language models are the engines behind many of the artificial intelligence tools we use today, from chatbots that answer questions to systems that help write code. These machines do not learn by being taught rules in a classroom; instead, they learn by reading vast amounts of text from the internet, books, and other digital sources. This process is known as training. For a model to become fluent and knowledgeable, it must consume a library of text so large that it would take a human lifetime to read it all. Researchers have long focused on how these models are built and how they are tuned after their initial learning to ensure they are safe and helpful. However, a critical question has remained largely unasked: exactly what kind of material is the model reading in the first place? If the training library contains harmful, hateful, or dangerous ideas, the model might learn to repeat them, even if its creators later try to teach it better behavior. This is not just a technical glitch; it is a matter of public safety, as these systems are increasingly used by people of all ages, including children, who might encounter dangerous viewpoints without warning.

A team of researchers at the University of Manchester set out to investigate the contents of the training data for one of the most significant open-source collections of text available to the scientific community. This collection, known as Dolma, serves as the foundation for a series of advanced language models called OLMo. The researchers wanted to know if this massive library contained extremist speech—text that promotes violence, hatred, or the overthrow of democratic systems. To answer this, they did not simply guess or rely on a single computer program. Instead, they built a careful, multi-step process to find these documents. They started by taking a random sample of 200,000 documents from the Dolma collection, which represents a tiny but statistically meaningful slice of the three trillion words the model was trained on. They then used several different computer models to scan these documents, asking each one to identify text that matched specific definitions of extremism. These definitions came from official government guidelines in the United Kingdom and from academic experts in the study of terrorism and political violence.

The process was designed to be conservative, meaning the researchers aimed to be very sure before labeling something as extremist. They used a technique called a "union check," where they asked the computer models to flag a document if it matched any of the different definitions of extremism. This helped ensure that the models were understanding the task correctly. Once the computers flagged potential matches, the researchers subjected them to further scrutiny. They used a more powerful artificial intelligence system to filter out false alarms, and finally, they brought in a human expert who specializes in the study of extremist speech to read a small sample of the flagged texts closely. This human review was crucial because the text found in the wild is often messy, filled with coded language, or written in a way that is difficult for machines to interpret. The expert read through these documents to confirm whether they truly contained the dangerous narratives the researchers were looking for.

The results of this investigation were clear and concerning. The researchers found that the Dolma training collection almost certainly contains hundreds of thousands of documents with extremist content. Based on their conservative estimates, they calculated that roughly one in every 2,000 documents in the random sample contained extremist narratives or hate speech. When they projected this rate back onto the entire dataset, which consists of billions of documents, the numbers pointed to a vast reservoir of harmful material. The study identified various types of dangerous content, including direct calls for violence, hate speech targeting specific racial or religious groups, and narratives that seek to undermine democratic institutions. Interestingly, the researchers found that the specific definition of extremism used to search the data changed the results. One definition, which focused on the conflict between "in-groups" and "out-groups," cast a very wide net and caught many texts that were hateful but not necessarily extremist in the political sense. Another definition, based on official government criteria, was more precise but still identified a significant amount of harmful material.

A striking discovery was the absence of a specific type of extremism in the sample. Despite the global nature of the internet, the researchers found almost no texts related to jihadist terrorism in their sample, with only a single news story about such an event appearing. This suggests that strict regulations and corporate policies regarding terrorist organizations may have successfully removed this specific type of content from the public datasets that feed these models. However, the study found that right-wing extremist narratives were far more prevalent, often appearing in the form of political blogs, forum comments, and conspiracy theories. The researchers also noted that much of the content they found was "borderline"—text that was highly polarized, violent, or hateful but did not always fit a strict legal definition of extremism. This gray area is particularly dangerous because it is harder to filter out, yet it can still normalize extreme views for the models that read them.

The implications of these findings extend beyond the technical details of how the models are built. The researchers argue that the problem cannot be solved simply by trying to fix the models after they have already learned from this data. If a model has memorized thousands of documents calling for violence or promoting hatred, it may struggle to unlearn those patterns, even with additional safety training. The study highlights that the composition of the training data is a fundamental issue that requires urgent attention from the people who build these systems. It is not enough to focus on how the models behave once they are released; the safety of the final product depends heavily on what was fed into it during its initial learning phase. The researchers conclude that there is a need for a broader discussion about what kinds of content are acceptable for inclusion in the training libraries of artificial intelligence, especially when those libraries are used to create tools that interact with the public every day.