💬 NLP

SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark

This paper introduces SomaliWeb v1, a publicly released, quality-filtered Somali corpus of approximately 303 million tokens accompanied by a matched BPE-16K tokenizer and the first public language-identification benchmark, which also exposes significant quality defects in existing multilingual Somali data distributions.

Original authors: Khalid Yusuf Dahir

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Khalid Yusuf Dahir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, chaotic library containing books in hundreds of different languages. For a long time, the "Somali" section of this library was a mess. It wasn't a dedicated room with a clear sign; instead, Somali books were scattered randomly inside giant, mixed-language stacks, often mixed with trash, broken pages, or duplicate copies.

SomaliWeb v1 is the first time someone has gone into that chaotic library, built a dedicated room just for Somali, and organized it properly. Here is how the authors did it, explained simply:

1. The Problem: The "Noisy" Library

Before this project, if you wanted to teach a computer to understand Somali, you had to grab a giant bag of mixed-language text from the internet.

The Mix: It contained Somali, but also English, French, and other languages mixed in.
The Noise: Even the "cleaned" versions of these bags were full of errors. The authors found that one popular bag (called HPLT v2) had:
- 17% duplicates: Like having the same book printed 17 times and stapled together.
- 56% "Mojibake": This is when text looks like gibberish (e.g., Ã© instead of é) because the computer read the letters wrong. It's like a book where the ink has smudged into nonsense.
- Near-duplicates: Books that are 90% the same, just with a few words changed.

2. The Solution: The Six-Stage "Sieve"

The authors built a six-step machine (a pipeline) to filter the raw internet text down to only the good stuff. Think of it like a gold panning process:

Step 1: The Duplicate Detector. They scanned every document and threw out exact copies. Result: They removed about 14% of the pile.
Step 2: The Fix-It Station. They used a tool to repair the "gibberish" text and threw out anything too short (less than 50 words). Result: They fixed half the remaining text and removed short, useless snippets.
Step 3: The Language Police. They ran a test to make sure the text was actually Somali. If a document was mostly English, it was kicked out. Result: A tiny bit of "English leakage" was removed.
Step 4: The "Almost-Clone" Hunter. They used a smart math trick (MinHash) to find documents that were 80% identical to each other and kept only the best version. Result: They removed another 10% of the pile.
Step 5: The Quality Check. They compared the text against a "gold standard" (Somali Wikipedia) to see if the words and patterns looked like fluent, natural Somali. If a document looked weird or broken, it was dropped. Result: They removed the bottom 15% of the lowest-quality text.
Step 6: The Final Polish. They shuffled the remaining documents, split them into a "training" set and a "testing" set, and created a custom tokenizer.

3. The New Tools: A Custom Dictionary

When computers read text, they break words into smaller pieces called "tokens."

The Old Way: Using a generic dictionary (like GPT-4's) meant Somali words were chopped into tiny, inefficient pieces. It was like trying to eat a large pizza with a tiny spoon; you need many, many spoonfuls (tokens) to finish the meal.
The New Way: The authors built a custom dictionary specifically for Somali.
- The Result: Their dictionary is 40% more efficient. It takes 40% fewer "spoonfuls" to read the same amount of text. This saves money and computing power for anyone using this data later.

4. The Benchmark: A "Driver's Test" for Language Detectors

The authors also created a test to see which computer program is best at identifying Somali text. They tested three popular tools:

The Surprise Winner: The oldest tool, called langdetect, performed the best at spotting Somali.
The Loser: A newer, more complex tool (fastText) failed miserably, often thinking Somali was a completely different language.
The Takeaway: Just because a tool is new doesn't mean it's better for specific languages.

Summary of What Was Released

The authors didn't just write a paper; they released three actual tools for the public:

The Corpus: A clean, high-quality collection of 819,000 Somali documents (about 303 million words).
The Tokenizer: A custom "dictionary" that reads Somali much more efficiently than generic ones.
The Benchmark: A public test scorecard showing which language detectors actually work for Somali.

What they did NOT do (according to the paper):
They did not train a new AI chatbot or language model on this data yet. They only built the ingredients (the clean data and the tools) and proved that the ingredients are high quality. They are saving the "cooking" (training a model and testing how smart it gets) for a future version (v2).

In short: SomaliWeb v1 is the first time someone has taken the messy, broken Somali text from the internet, cleaned it up, organized it, and handed it to researchers with a custom set of tools, so they can finally build better AI for Somali speakers without having to do all the cleaning themselves.

Technical Summary: SomaliWeb v1

Problem Statement
Despite Somali being a major world language with approximately 25 million speakers and a vibrant digital ecosystem (news, diaspora, social media), it lacks a dedicated, documented, single-language pretraining corpus. Existing Somali text is fragmented within multilingual distributions (e.g., HPLT v2, CC100, MADLAD-400, mC4) or exists as small, undocumented uploads on Hugging Face. This infrastructure gap prevents practitioners from auditing training data quality and forces researchers to repeatedly re-derive corpora from raw multilingual dumps, rediscovering the same quality issues. Furthermore, no public benchmark exists for language identification (LID) specifically tuned for Somali, and no matched tokenizer has been released to address the "tokenization tax" (inefficient token counts) faced by low-resource languages in English-centric models.

Methodology: The Six-Stage Pipeline
The authors present SomaliWeb v1, constructed via a reproducible, six-stage pipeline aggregating three upstream sources: HPLT v2 (som_Latn), CC100 (so), and the Somali Wikipedia (2023-11-01 dump). The pipeline processes an initial aggregate of 1,372,052 documents (~~588M tokens) down to a final corpus of 819,322 documents (~~303M tokens).

Byte-Exact Deduplication: Documents are hashed (SHA-256) after lowercase normalization and whitespace collapse. This removes 13.83% of the raw input, revealing that HPLT v2 alone contained 17.27% byte-exact duplicates.
Normalization and Length Filtering: A sequence of operators repairs mojibake (using ftfy), applies Unicode NFC normalization, collapses whitespace, and removes repeated character runs. Documents with fewer than 50 words are discarded. This stage fixes mojibake in 56.06% of surviving HPLT documents and drops 8.65% of the corpus (primarily short documents).
Language Identification (LID): Seeded langdetect filters for documents where Somali is the top-1 prediction with confidence $\ge$ 0.50. A secondary pass with GlotLID v3 tags dialects. This removes 0.21% of documents, mostly English leakage.
Near-Duplicate Removal: MinHash with Locality Sensitive Hashing (LSH) identifies near-duplicates using word-3-grams. Candidates are verified with exact Jaccard similarity ( $\tau = 0.80$ ). Clusters are merged, retaining the longest document. This removes 10.57% of documents (113,896 documents).
Quality Filtering: Documents are scored based on character-5-gram coverage against a clean seed of 2,221 Somali Wikipedia articles. The bottom 15% of documents by coverage are dropped. This step reveals source-quality asymmetry, with Wikipedia dropping 20.61% of its documents compared to 5.76% for CC100.
Release and Tokenizer: The final corpus is shuffled, split 95/5 (train/validation), and used to train a BPE-16K tokenizer.

Key Contributions

C1 (Artifact): The first versioned, documented, single-language Somali pretraining corpus (819k documents, ~303M tokens) released with a companion tokenizer and a full dataset card under CC-BY-SA 4.0.
C2 (Benchmark): The first public side-by-side benchmark of three production LID tools (langdetect, GlotLID v3, fastText lid.176) on a Somali-specific test set, providing per-class precision, recall, F1, and throughput with 95% bootstrap confidence intervals.
C3 (Measurement): A quantified audit of HPLT v2's "cleaned" Somali distribution, identifying three concrete defects: 17.3% byte-exact duplicates, 56.1% of documents containing fixable mojibake, and 10.7% near-duplicates among byte-unique documents.
C4 (Tool): A BPE-16K tokenizer trained on the cleaned corpus. It achieves 40.2% higher token efficiency (fewer tokens per word) on the FLORES-200 Somali devtest compared to GPT-4's cl100k_base and matches the fertility of a tokenizer trained on the raw HPLT corpus despite using a 30% smaller training set.

Results

Pipeline Retention: The pipeline retains 59.7% of the raw aggregated input. The most significant drops occur in Phase 1 (deduplication) and Phase 5 (quality filtering).
LID Performance: On a 200-row test set, langdetect achieved the highest Somali F1 score (0.884), outperforming GlotLID v3 (0.829) and fastText lid.176 (0.140). Notably, fastText failed to identify most Somali documents (recall 0.075), while GlotLID frequently confused Somali with phylogenetically distant Latin-script African languages (e.g., Oromo, Fulfulde) rather than the related Maay Maay dialect.
Tokenizer Efficiency: The SomaliWeb v1 tokenizer emits 35,867 tokens for the FLORES-200 devtest, compared to 60,010 tokens for cl100k_base. This represents a 1.67x reduction in token count, implying a potential reduction in inference costs and a higher information density per token.
Dialect Coverage: The final corpus contains zero documents identified as Maay Maay (ymm_Latn), likely due to the oral nature of the dialect and upstream source filtering. The corpus is explicitly labeled as "Standard Somali" only.

Significance and Claims
The paper frames SomaliWeb v1 primarily as an audit of an artifact rather than a claim of downstream language modeling superiority. Its significance lies in:

Reproducibility: Providing a documented, bit-exactly reproducible pipeline that allows the community to audit and iterate on Somali data curation.
Quality Awareness: Demonstrating that "available" multilingual data is not necessarily "clean," and that significant quality defects (duplicates, mojibake) persist in major releases like HPLT v2.
Tokenization Efficiency: Establishing that a corpus-matched tokenizer can significantly reduce the tokenization tax for Somali without requiring larger model architectures.
Foundation for Future Work: The authors explicitly state that downstream language-model perplexity comparisons are deferred to a future release (v2), which will include ablation studies and baseline comparisons against other pipelines (e.g., CCNet).

The work is modest in its claims, acknowledging limitations such as the lack of PII scrubbing (Somali-specific), the heuristic nature of the quality filter, the small size of the LID test set, and the exclusion of non-standard dialects. It positions itself as the first step in a versioned effort to build robust NLP infrastructure for the Horn of Africa.