Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
This paper introduces Sampled-BPE, a lightweight token-level auditing pipeline that efficiently identifies pollution in web-scale Chinese corpora by training BPE tokenizers on small samples, revealing widespread and shifting contamination across open datasets while providing a hierarchical dataset for further analysis.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a giant, endless library where every book, blog, and broken webpage is a single page of text. For years, scientists have been trying to teach computers to read and understand this library to build "Large Language Models" (LLMs)—super-smart AI assistants like the ones you might chat with. But here's the problem: the internet isn't just a clean library; it's also a messy, chaotic marketplace filled with spam, scams, and inappropriate content. When AI learns from this messy library, it can accidentally pick up bad habits, like learning to write about online gambling or generating strange, offensive phrases. This is called "corpus pollution." To fix it, researchers need to audit the library, but the library is so huge (trillions of pages!) that reading every single page would take longer than a human lifetime. They need a way to peek inside and find the bad apples without checking every single one.
This is exactly what the paper "Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics" tackles. The authors, a team from Tsinghua University and other institutions, realized that trying to scan the entire Chinese internet for bad content was too slow and expensive. So, they invented a clever shortcut called SAMPLED-BPE. Think of it like a marine biologist who wants to know how polluted a massive ocean is. Instead of scooping up every drop of water, they take a few small samples from different spots, analyze them, and use that to estimate the pollution level of the whole ocean. In this case, the "ocean" is the Chinese web, and the "pollution" is toxic or spammy words (tokens) that get baked into AI models.
The team's method is surprisingly simple but powerful. They take a tiny slice of the massive Chinese text data (as little as 0.25% of the total), train a special "tokenizer" (a tool that breaks text into small chunks called tokens) on just that slice, and then look at which tokens show up the most often. If a chunk of text appears constantly in their small sample, it's likely a common pattern in the whole dataset. They then use a smart AI assistant to check what those frequent chunks actually mean by searching the web. If a chunk turns out to be a gambling site or a pornographic phrase, they flag it.
Their findings are a bit of a shocker. They audited 11 different open-source Chinese text collections and 6 snapshots of the Chinese web from 2021 to 2026. They found that pollution is everywhere, but it's not spread out evenly. Some datasets are relatively clean, while others are absolutely drowning in bad content. For instance, one dataset called OSCAR was found to be over 83% polluted, with the majority of that being adult content. Another, mC4, was heavily polluted with online gambling terms. Even more interestingly, they discovered that the "Chinese Common Crawl" (a massive, raw dump of the web) is highly polluted, and the type of pollution changes over time. In 2026, nearly 69% of the content in their snapshot was adult material, whereas gambling content had dropped significantly since 2021. This suggests that the bad stuff on the web is a moving target; as soon as one type of spam is blocked, another pops up.
The paper also argues against the idea that you can just make a fixed list of "bad words" to filter out. Because the pollution changes so fast and often uses sneaky, abbreviated, or combined words (like mixing two normal words to make a gambling term), a static list would quickly become useless. Instead, the authors suggest that we need to keep auditing the web regularly, just like they did. To help others do this, they released a massive new dataset containing over 630,000 records of these polluted tokens, organized into "trees" that show how short, bad words grow into longer, more complex bad phrases.
In short, the paper proves that you don't need to read the whole ocean to know it's dirty; a smart, small sample is enough to get a clear picture. They showed that the Chinese web is currently a very polluted place for AI training, with the pollution shifting and evolving rapidly. Their new tool, SAMPLED-BPE, makes it possible to check these massive datasets in hours instead of months, giving researchers a way to keep AI models clean without getting overwhelmed by the sheer size of the internet.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.