MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages

1. Problem Statement

The paper addresses the scarcity of decoder-only language models specifically designed for South African languages, particularly the nine low-resource Bantu languages among the country's eleven official written languages.

• Current Landscape: Existing research for African languages has historically relied on encoder-only (e.g., BERT) or encoder-decoder (e.g., T5) architectures. While decoder-only models are gaining traction, most cover only subsets of languages (e.g., isiZulu, isiXhosa, Afrikaans) or rely on massive parameter counts (e.g., LLaMA-3-8B) that are computationally expensive and difficult to reproduce in low-resource settings.
• The Gap: There is no publicly available, open-weight decoder-only model trained from scratch that covers all eleven official South African languages. Furthermore, it is unclear which adaptation strategy (monolingual vs. multilingual vs. instruction tuning) is most effective for small-scale models (e.g., <1B parameters) in low-resource contexts.

2. Methodology

A. Data Construction: MzansiText

The authors created MzansiText, a curated multilingual pretraining corpus covering all eleven official languages.

• Sources: Aggregated from public datasets including mC4, CulturaX, WURA, Glot500-c, NCHLT Text, CC100, ParaCrawl, and Inkuba-Mono.
• Preprocessing Pipeline: A deterministic, multi-stage pipeline using Datatrove was implemented to ensure reproducibility. Steps included:
  1. Language identification (retaining only the 11 target languages).
  2. Normalization and boilerplate removal (HTML stripping, Unicode normalization).
  3. Structural constraint enforcement (length, script).
  4. Deduplication (exact and near-duplicate removal via MinHash).
  5. Safety screening (filtering PII and sensitive content).
  6. Quality filtering (removing templated/low-info text).
  7. Packing into ~1k token training records.
• Distribution: The dataset is heavily skewed toward high-resource languages (Afrikaans and English comprise ~84% of tokens). The authors experimented with balancing the distribution but found it degraded downstream performance; thus, they retained the natural distribution to maximize usable data volume while ensuring coverage for all languages.
• Tokenization: A BPE tokenizer with a vocabulary size of 65,536 was trained on the pretraining split.

B. Model Architecture: MzansiLM

• Architecture: A compact decoder-only model trained from scratch.
• Scale: 125 million parameters, utilizing the MobileLLM architecture configuration optimized for small-scale models.
• Design: Employs a LLaMA-style decoder stack with causal self-attention.
• Training: Trained for 5 epochs on the MzansiText training split using 4 NVIDIA A100 GPUs. The context length is 2,048 tokens. Loss is averaged across languages to prevent high-resource languages from dominating the validation signal.

C. Adaptation Regimes (Instruction Finetuning)

The authors systematically evaluated three adaptation strategies to determine the optimal approach for small-scale models:

1. Monolingual Task-Specific Finetuning: Training a separate model for each specific task-language pair.
2. Multilingual Task-Specific Finetuning: Training one model per task by pooling data from all languages for that specific task (enabling cross-lingual transfer).
3. General Multi-Task Instruction Finetuning: Training a single model on a weighted mixture of all tasks and languages simultaneously.

3. Key Contributions

1. MzansiText: The first curated, reproducible pretraining corpus covering all eleven official written South African languages with a transparent filtering pipeline.
2. MzansiLM: A 125M-parameter decoder-only model trained from scratch, released as an open-weight baseline.
3. Systematic Evaluation: A comprehensive comparison of adaptation strategies (monolingual, multilingual, and multi-task) across Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks.
4. Resource Release: Public release of the dataset, model, and code for data preparation, pretraining, and evaluation.

4. Results and Findings

The model was evaluated on tasks including document classification, named entity recognition (NER), part-of-speech tagging (POS), data-to-text generation, headline generation, and few-shot reasoning.

Performance by Task Type:

• Data-to-Text Generation (T2X):
  • Monolingual task-specific finetuning was the clear winner.
  • MzansiLM achieved 20.65 BLEU on isiXhosa, outperforming the 0.58B parameter mT5-base and competing with encoder-decoder models over 10x larger (e.g., Aya-13B).
• Topic Classification (MasakhaNEWS):
  • Multilingual task-specific finetuning performed best for closely related Bantu languages (e.g., 78.5% macro-F1 on isiXhosa), leveraging shared signals across languages.
  • Monolingual finetuning was competitive for English.
• Sequence Labelling (NER & POS):
  • Monolingual task-specific finetuning was the only effective regime. Base models and general instruction tuning performed near chance.
  • MzansiLM (125M) with monolingual finetuning substantially outperformed larger decoder-only baselines (including 8B and 70B parameter models) on NER, though it still trailed behind encoder-only models (e.g., AfroXLMR-large).
• Few-Shot Reasoning & Reading Comprehension:
  • Poor Performance: On tasks like intent classification (INJONGO), reading comprehension (Belebele), and reasoning benchmarks (IrokoBench), MzansiLM performed near chance levels.
  • Scaling Limit: Performance on these tasks remained stagnant even for decoder-only models up to 8B parameters. Significant gains were only observed at the 70B parameter scale (e.g., Llama-3.1-70B) or with encoder-decoder architectures (e.g., Aya-101).

Key Comparative Insights:

• Decoder-Only vs. Encoder-Only: Encoder-only models remain superior for structured prediction tasks (classification, NER) and high-accuracy requirements.
• Decoder-Only vs. Encoder-Decoder: Encoder-decoder models dominate reading comprehension and headline generation.
• Small-Scale Viability: Despite its small size, MzansiLM proves that task-specific supervision allows small decoder-only models to outperform much larger models on specific generation and sequence labelling tasks, provided the adaptation strategy is tailored (monolingual for generation/NER, multilingual for classification).

5. Significance and Conclusion

• Reproducibility: The paper establishes a reproducible baseline for South African language modeling, demonstrating that high-quality results can be achieved with small models (125M) if the data and adaptation strategies are optimized.
• Strategic Guidance: It clarifies that for low-resource languages, task-specific finetuning is critical. "General" instruction tuning is less effective at small scales for specific tasks.
• Architectural Trade-offs: The study highlights that while decoder-only models are excellent for generative tasks with targeted supervision, encoder-based architectures remain the preferred choice for classification and structured prediction in low-resource settings until decoder-only models scale significantly (likely >70B).
• Limitations: The model struggles with emergent capabilities like few-shot reasoning and long-range dependencies due to its small parameter count and the skewed data distribution (dominated by English/Afrikaans).

In summary, MzansiLM serves as a vital, accessible baseline that helps researchers identify where small decoder-only models can succeed (targeted generation/NER) and where alternative architectures or larger scales are still necessary.