💬 NLP

PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

This paper introduces PashtoCorp, a 1.25-billion-word corpus and reproducible pipeline for Pashto that significantly outperforms previous datasets, leading to substantial improvements in language model perplexity, named entity recognition, and reading comprehension benchmarks for this low-resource language.

Original authors: Hanif Rahman

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanif Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to speak Pashto, a language spoken by 60 million people. Until now, the robot's "library" of Pashto books was tiny—like trying to learn a whole language from a single pamphlet. The robot could barely understand anything, and it made mistakes constantly.

This paper introduces PashtoCorp, a massive new project that changes the game. Here is the story of what they did, explained simply:

1. The Problem: A Thirsty Library

Think of the Pashto language as a giant, vibrant city. But for computers (AI), this city was mostly empty. The biggest existing collections of Pashto text were like a few small bookshelves in a massive library. They were too small to teach a smart robot anything useful.

2. The Solution: Building a "Super-Library"

The researchers built PashtoCorp, a digital library containing 1.25 billion words.

How big is that? It's 40 times bigger than the previous biggest collection and 83 times bigger than the old record-holder.
Where did the books come from? They didn't just copy-paste one thing. They built 32 custom "robot spiders" that crawled the internet to gather text from:
- News sites (like BBC Pashto or local Afghan news).
- Radio transcripts (what people actually say on the air).
- PDFs of books and government reports (the "deep knowledge" stuff).
- Wikipedia (the encyclopedia).
- Existing internet datasets.

3. The Cleaning Process: The "Gold Panner"

You can't just dump a mountain of dirt and expect to find gold; you have to sift it. The researchers built a pipeline to clean the data:

Language Check: They threw away anything that wasn't actually Pashto (like English or Urdu mixed in).
Duplicate Removal: If the same article appeared on five different websites, they kept only one copy.
Quality Control: They tossed out tiny, broken snippets of text that were too short to be useful.
Result: They kept the "gold" (high-quality text) and threw away the "dirt."

4. The Experiment: Teaching the Robot

They took a standard AI model (a robot that knows many languages but is bad at Pashto) and fed it this new library.

The Result: The robot got 25% smarter at understanding Pashto grammar and structure.
The "Name" Test: They tested the robot on a task called "Named Entity Recognition" (finding names of people, places, and organizations).
- Before: The robot was guessing wildly and was very inconsistent (like a student who gets an A one day and an F the next).
- After: The robot became much more accurate and 7 times more consistent. It stopped guessing and started knowing.

5. The Big Surprise: It's Not About How Much, It's About What

This is the most interesting part of the paper. The researchers did a "leave-one-out" test. They asked: "What happens if we remove a specific type of source?"

The News/Radio (The Bulk): These made up 35% of the library. Removing them hurt the robot a little, but it was still okay.
The Wikipedia (The Tiny Slice): Wikipedia was only 0.7% of the total library (a tiny crumb). But when they removed it, the robot's ability to find names crashed by 47%.
The PDF Books (The Tiny Slice): These were only 0.6% of the library. But they contained 35% of all the unique words in the entire collection.

The Analogy: Imagine trying to learn a language. Reading 1,000 identical newspaper headlines (News/Radio) helps you learn common words like "the," "and," and "today." But reading one single, dense encyclopedia entry (Wikipedia) or a complex book (PDFs) teaches you the hard words, the specific names, and the deep concepts. A little bit of high-quality, diverse text is worth more than a mountain of repetitive text.

6. Where It Still Struggles

The robot got great at understanding news and formal writing. However, when they tested it on social media slang (like Twitter or Facebook posts full of abbreviations and informal speech), the robot didn't improve.

Why? Because the library didn't have enough "street slang." It was mostly formal news. This teaches us that if you want an AI to understand your text messages, you need to feed it text messages, not just news articles.

7. The Takeaway for the Future

The authors released everything for free: the library, the trained robot brain, and the code.

The Lesson: If you want to build AI for a language that doesn't have much data, don't just scrape the whole internet. Focus on finding encyclopedias, books, and high-quality articles first. Even a small amount of these "dense" texts will teach the AI more than a huge pile of repetitive news.

In short: They built a massive, clean library for Pashto, proved that a little bit of "smart" text is better than a lot of "noisy" text, and gave everyone the keys to the library so they can build better tools for the 60 million Pashto speakers.

1. Problem Statement

Natural Language Processing (NLP) has advanced rapidly, but progress remains uneven across languages. Pashto, an Indo-Iranian language spoken by approximately 60 million people in Afghanistan, Pakistan, and global diaspora communities, is severely underrepresented.

Data Scarcity: Prior to this work, the largest available resources were the OSCAR 2301 subset (~~31M words) and the NLPashto corpus (~~15M words).
Limitations: These existing datasets are too small for competitive language model training. Even the Pashto Wikipedia contains only ~6M words.
Consequence: The lack of large-scale, high-quality data hinders the development of robust foundation models and downstream applications for Pashto speakers.

2. Methodology

The authors developed a fully reproducible pipeline to construct PashtoCorp, a 1.25-billion-word corpus.

A. Data Collection

The corpus aggregates data from 39 heterogeneous sources:

7 HuggingFace Datasets: Including FineWeb2, HPLT v2.0, CulturaX, CC-100, MADLAD-400, and GlotCC.
32 Custom Web Scrapers: Built using Scrapy to target Pashto news outlets and radio services (e.g., Azadi Radio, VOA Pashto, BBC Pashto, Pajhwok Afghan News).
Volume: The final corpus comprises 2.81 million documents totaling 1.25 billion words.

B. Processing Pipeline

The pipeline ensures quality and reproducibility through three main stages:

Language Identification: Documents are scored based on the fraction of tokens falling within Pashto Unicode ranges (U+0600–U+06FF). Documents with <70% Pashto script tokens are discarded to remove code-switched or mislabeled text while retaining proper nouns and numerals.
Deduplication: Uses SHA-256 hashing on lowercased, whitespace-normalized content to remove duplicates.
Quality Filtering: Removes documents with fewer than 10 tokens.

Result: The pipeline rejected ~25.5% of raw data, retaining 74.5% for the final corpus.

C. Evaluation Suite

The authors established a comprehensive evaluation framework including:

Intrinsic Metrics: Perplexity (PPL) and vocabulary coverage.
Extrinsic Tasks:
- WikiANN: Named Entity Recognition (NER) with fixed train/validation/test splits.
- POLD: Offensive language detection on social media.
- Belebele: Reading comprehension (multiple-choice).
Ablation Studies: Leave-one-out experiments to quantify the contribution of specific domain groups (e.g., Wikipedia, News, PDFs) to model performance.

3. Key Contributions

PashtoCorp: A massive, 1.25B-word corpus that is 40× larger than the OSCAR Pashto subset and 83× larger than the previous largest dedicated Pashto corpus.
Reproducible Pipeline: A modular, open-source pipeline capable of adapting to other Arabic-script languages (e.g., Dari, Urdu) with minimal configuration changes (only Unicode range adjustments).
Comprehensive Evaluation Suite: The first standardized benchmarks for Pashto, including the first published LLM baseline for the Belebele reading comprehension task.
Source Ablation Analysis: Quantitative evidence showing that domain diversity is more critical than raw volume for specific tasks.

4. Key Results

A. Corpus Scale and Diversity

Vocabulary: The corpus contains 6.32 million unique word types.
Source Impact: Despite comprising only 0.6% of the total documents, PDF books/reports contributed 35% of the unique vocabulary. Conversely, news radio (35.6% of docs) contributed only 0.9% of unique vocabulary due to lexical homogeneity.

B. Intrinsic Performance (Language Modeling)

Perplexity Reduction: Continued pretraining of XLM-R-base on PashtoCorp reduced held-out perplexity by 25.1% (from 8.08 to 6.06).
Sample Efficiency: The pretrained model showed the largest relative gain in NER tasks when training data was scarce (at 50 sentences, a +27% F1 improvement).

C. Extrinsic Performance

Named Entity Recognition (WikiANN):
- F1 Score: Improved from 19.0% (baseline) to 21.0% (+10% relative).
- Stability: Training variance decreased nearly 7× (from ±4.7% to ±0.7%), indicating more robust learning.
- Vocabulary Coverage: PashtoCorp covers 97.9% of WikiANN entity tokens (100% for Person names).
Offensive Language Detection (POLD):
- Null Result: The pretrained model showed no improvement over the baseline.
- Reason: A domain gap exists; PashtoCorp is dominated by formal news/text, while POLD consists of colloquial social media. The corpus covered only 91.6% of POLD vocabulary types.
Reading Comprehension (Belebele):
- Gemma-3n-E4B achieved 64.6% accuracy, establishing the first published LLM baseline for Pashto on this benchmark.

D. Source Ablation Insights

Wikipedia: Although only 0.7% of the documents, removing Wikipedia caused a 47% relative drop in NER F1 scores, making it the most critical source for entity recognition.
Web Crawl: Removing general web crawl data had the largest negative impact on general language model quality (Perplexity rose from 6.06 to 6.58).

5. Significance and Implications

Domain Alignment is Critical: The study demonstrates that pretraining benefits downstream tasks only when the corpus domain aligns with the task domain (e.g., News/Wikipedia $\to$ NER; Social Media $\to$ POLD).
Value of Small, Diverse Sources: The research challenges the "volume-first" approach. Small, high-quality sources like PDFs and Wikipedia provide disproportionate value for vocabulary richness and specific tasks compared to massive, repetitive web crawls.
Reproducibility for Low-Resource Languages: The pipeline is designed to be easily transferable to other under-resourced languages (Dari, Urdu, Sindhi) with minimal code changes, offering a blueprint for the community.
Ethical Considerations: The authors highlight biases in the corpus (over-representation of Afghan/Kandahari dialects and formal registers) and warn against using the data for surveillance of vulnerable Pashto-speaking communities.

Conclusion

PashtoCorp represents a significant leap forward for Pashto NLP, providing a scalable, high-quality resource that enables the training of more robust language models. The accompanying evaluation suite and ablation studies offer actionable guidance for practitioners: prioritize domain diversity and encyclopedic sources over raw volume when building corpora for low-resource languages. All data, models, and code are publicly available to foster further research.