Wiki Dumps to Training Corpora: South Slavic Case
This paper presents a language-agnostic methodology for transforming raw Wikimedia dumps into high-quality, linguistically rich training corpora for seven South Slavic languages by systematically extracting text from multiple wiki projects and employing n-gram-based filtering to remove low-quality, repetitive articles.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you want to build a massive library of stories, poems, and news articles to teach a robot how to speak and understand seven different South Slavic languages (like Serbian, Croatian, Bulgarian, and others). You decide to use Wikipedia and its sister sites (like Wikisource or Wikinews) as your source material because they are free and full of text.
However, if you just grab the raw files from these websites, you aren't getting a library of stories. You're getting a chaotic warehouse filled with:
- Construction debris: Hidden computer codes, formatting instructions, and "blueprints" that tell the website how to look but aren't part of the story.
- Broken glass: Broken links and empty sections.
- Factory-made clones: Thousands of articles that look almost identical because they were automatically generated from databases rather than written by humans.
This paper is a guide on how to clean up that messy warehouse and turn it into a pristine library. The author, Mihailo Škorić, breaks the process down into two main phases: The Great Cleanup and The Quality Filter.
Phase 1: The Great Cleanup (Text Extraction)
Think of the raw Wikipedia data as a house that is still under construction. It has scaffolding, paint cans, and blueprints everywhere. You can't live in it yet.
The author built a specialized "construction crew" (a computer program) to strip the house down to its bare, livable walls.
- Stripping the Scaffolding: The program uses a tool called
mwparserfromhell(a fancy name for a tool that understands Wikipedia's secret language) to rip out all the computer code, templates, and formatting tags. - Removing the Noise: It deletes the "Category" lists (like tags on a file cabinet), strips out image descriptions, and removes the "References" sections that look like footnotes but aren't part of the main story.
- Flattening the Tables: Wikipedia often uses tables to organize data. The program turns these complex grids into simple, readable sentences so the robot doesn't get confused by the grid lines.
- The Result: After this phase, you have a pile of clean, plain text. It's no longer a website; it's just words.
Phase 2: The Quality Filter (Removing the Clones)
Now that you have clean text, there's a new problem. Some of the articles are "zombie" texts. These are articles that look like they were written by a human, but they were actually auto-generated by a computer. They are repetitive, boring, and say the same thing over and over in slightly different ways.
If you feed these "zombie" articles to your robot, it will learn to speak in a robotic, repetitive loop. To fix this, the author uses a Detective Strategy:
- Grouping by Neighborhood: The program groups articles by their topic (e.g., all articles about "History" go in one room, all about "Sports" in another). It's easier to find duplicates if you only compare apples to apples.
- The "Fingerprint" Scan: The program turns every article into a mathematical "fingerprint" (a vector). It looks at the first few words of the article to see if it matches the pattern of a template.
- The Similarity Test: It compares these fingerprints against each other using a math trick called MinHashing. Imagine you have two books. If they share too many identical sentences in the same order, they are likely clones.
- The Cutoff: The program calculates a "similarity score." If an article is too similar to others (above a certain threshold), it gets kicked out of the library. It's like a bouncer at a club who says, "You look exactly like everyone else in line; you're not getting in."
The Results
The paper tested this method on seven South Slavic languages. The results were dramatic:
- Serbian: Had the biggest mess to clean. Before filtering, it had over 500,000 articles. After the "bouncer" did their job, nearly half were removed because they were duplicates or auto-generated. The word count dropped by nearly 60%.
- Bulgarian and Slovenian: These were already quite clean, so they lost fewer articles.
- The Payoff: The final result is a smaller, but much higher-quality library. The words in these new libraries match how real humans actually speak, rather than how a database generates text.
Why This Matters
The author argues that for a robot to learn a language well, it needs to read real human writing, not computer-generated filler. By doing this two-step process (cleaning the code, then filtering the clones), the paper provides a reliable, high-quality set of books for training AI models in South Slavic languages.
In short: The paper teaches us how to take a messy, code-filled dump of Wikipedia, scrub it clean of computer junk, and then throw out the "copy-paste" articles, leaving behind a pure collection of human language ready for a robot to learn from.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.