💬 NLP

Sample-Size Scaling of the African Languages NLI Evaluation

This study reveals that increasing labeled data for Natural Language Inference across 16 African languages does not guarantee consistent performance gains, as results often exhibit non-monotonic, language-specific scaling behaviors that necessitate tailored dataset creation and advanced multilingual modeling strategies.

Original authors: Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion, Jesujuwon Egbewale, Hannah Nwokocha

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion, Jesujuwon Egbewale, Hannah Nwokocha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge how well a student understands a subject, but instead of giving them a full exam, you only show them a few random questions. You might think, "The more questions I ask, the better I'll know what they really know."

This paper is about testing that exact idea, but instead of a student, it's about AI models trying to understand African languages. The researchers wanted to see if giving these AI models more "practice questions" (labeled data) always makes them smarter, or if the relationship is more complicated.

Here is the breakdown of their findings using simple analogies:

1. The Big Misconception: "More Data = Always Better"

In the world of high-tech AI, there is a common belief that if you just keep feeding a computer more examples, it will get better and better, like a muscle that grows stronger with every workout.

The Paper's Reality Check:
The researchers found that for African languages, this isn't always true. Sometimes, adding more data actually makes the AI's performance score go down or stay the same. It's not a straight line up; it's a messy, bumpy road that changes depending on which language you are talking about.

2. The "Small Sample" Trap (The Lucky Guess)

When the researchers tested the AI with very few examples (like 50 questions), the results were often too good to be true.

The Analogy: Imagine a dart player who throws 3 darts and hits the bullseye every time. You might think, "Wow, they are a pro!" But if they throw 500 darts, you might realize they were just lucky with those first three.
The Finding: With small groups of data, the AI looks incredibly smart because it's only seeing the "easy" or "obvious" examples. This is called "small-sample optimism." The researchers found that as they added more questions (up to 300 or 400), the AI's score often dropped. This didn't mean the AI got worse; it meant the test was finally showing the AI's real ability, including the hard parts it couldn't guess.

3. Every Language is a Different Puzzle

The most surprising discovery was that not all languages behave the same way.

The Analogy: Think of different languages as different types of terrain.
- Language A (like Yoruba in the study): Imagine a hill that looks steep and easy to climb at the bottom, but as you go higher, the path gets slippery and you slide back down. The AI's score actually got worse as they added more data.
- Language B (like Kinyarwanda): Imagine a hill that gets steeper for a while, then flattens out into a plateau. The AI's score went up a bit, then stopped changing.
- Language C (like Wolof): Imagine a hill that is so tall you can't even see the top after 500 steps. The AI's score was still wobbling and unstable even after a lot of testing.

The researchers found that the "shape" of the learning curve depends entirely on the specific language, not just the AI model itself.

4. The "Sweet Spot" for Testing

The paper tried to figure out: How many questions do we need to ask to get a reliable answer?

The Finding: There is no single magic number for all languages.
- For some languages, 200 questions were enough to get a stable, reliable score.
- For others, you needed 450 or even 500 questions.
- For one language (Wolof), even 500 questions wasn't enough to get a stable score; the results were still too shaky.

5. What This Means for the Future

The researchers are essentially saying: "Stop trusting a single test score from a tiny group of examples."

If you test an AI on a small group of African language speakers, you might get a score that looks great, but it's a lie. To get the truth, you need:

More data: At least 300 examples to smooth out the "luck."
Repetition: Test the same language multiple times with different groups of people to see if the score holds up.
Patience: Accept that some languages are just harder to test than others and require more effort to get a fair grade.

Summary

The paper argues that in the world of African languages, more data doesn't always mean a straight line to success. Sometimes, more data reveals that the AI was just guessing well before. To truly understand how smart an AI is in these languages, we need to stop using tiny, lucky samples and start using larger, more careful tests that respect the unique quirks of each language.

Technical Summary: Sample-Size Scaling of the African Languages NLI Evaluation

Problem Statement

The paper addresses a critical gap in low-resource Natural Language Processing (NLP): the assumption that increasing the volume of labeled data invariably leads to improved downstream performance. While this "monotonic scaling" law holds for high-resource languages, its validity for African languages remains unverified. Given the high cost of annotation and the typological diversity of African languages, blindly expanding datasets without understanding scaling behavior risks inefficiency or counterproductive outcomes. The study specifically investigates whether increasing labeled data reliably enhances Natural Language Inference (NLI) performance across 16 African languages and whether evaluation stability is achievable with standard sample sizes.

Methodology

The authors conducted a systematic sample-size scaling study using the AfriXNLI benchmark, a human-translated version of the XNLI benchmark for African languages.

Dataset: Experiments were performed on 16 African languages representing diverse language families, scripts, and typological properties.
Models: Two multilingual transformer models with approximately 0.6 billion parameters were evaluated to control for architecture and scale:
1. XLM-R Large: Fine-tuned on the standard XNLI dataset.
2. AfroXLM-R Large: An African-centric variant of XLM-R trained with a focus on African language data.
Experimental Design:
- Sample Size Variation: Models were tested on sample sizes ranging from 50 to 500 labeled examples.
- Subsampling: To control for variance, multiple random subsampling runs were executed for each sample size.
- Metrics: Primary metrics included accuracy, precision, and F1-score. The study focused on the stability of these metrics (mean and standard deviation) rather than training dynamics, as models were not retrained on increasing data but evaluated on varying test set sizes.
- Saturation Analysis: The authors defined a saturation point ( $n^*$ ) as the minimum sample size where the mean accuracy varies by no more than $\pm0.5\%$ upon further data addition.

Key Results

1. High Variance in Low-Resource Regimes

Evaluation variance (standard deviation of accuracy) is highest at low sample sizes (50–100 examples) and decreases sharply as sample size increases, stabilizing around 300 samples. This indicates that performance estimates derived from small test sets are highly unstable and dependent on the specific samples chosen.

2. Non-Monotonic and Language-Specific Scaling

Contrary to the belief that more data always yields better performance, the study found strongly language-sensitive and often non-monotonic scaling behavior:

Monotonic Degradation: Some languages (e.g., Yoruba) exhibited a consistent decline in accuracy as sample size increased. This suggests that small subsets suffer from "small-sample optimism," overestimating performance by missing difficult instances, neutral cases, or translation ambiguities that appear in larger, more representative sets.
Saturation and Decline: Other languages (e.g., Kinyarwanda) showed initial gains followed by saturation or slight decline.
Heterogeneity: The scaling slopes varied significantly across languages. Some showed near-zero or negative slopes, indicating that increasing evaluation set size can reveal systematic errors rather than improve model capability.

3. Model Independence of Scaling Trends

While AfroXLM-R Large generally achieved higher initial accuracy and lower variance at very small sample sizes compared to XLM-R Large, both models exhibited similar qualitative scaling behaviors across languages. This implies that scaling behavior is driven more by language-specific and dataset-specific factors (e.g., morphology, translation quality) than by the pretraining strategy or model architecture.

4. Variable Saturation Points

The required sample size to achieve stable performance estimates varies drastically between languages:

Low Saturation: Languages like Shona, Yoruba, and Xhosa stabilized around 200–250 samples.
High Saturation: Languages like Swahili, Kinyarwanda, and Oromo required 400–450 samples.
Unstable: Wolof (under XLM-R Large) did not stabilize even at 500 samples, suggesting inherent instability in performance estimation for this language under current conditions.

Key Contributions

Empirical Characterization of Scaling: The paper provides the first systematic analysis of sample-size scaling for NLI in African languages, challenging the universal applicability of monotonic scaling laws.
Identification of Evaluation Bias: It demonstrates that small test sets systematically overestimate model performance in African NLI due to high variance and the exclusion of ambiguous or difficult instances.
Language-Specific Saturation Metrics: The authors provide estimated saturation points ( $n^*$ ) for 16 languages, highlighting that a "one-size-fits-all" test set size is insufficient for reliable benchmarking.
Methodological Recommendations: The study argues for shifting focus from single-point accuracy scores to variance-aware reporting and language-specific evaluation sizes.

Significance and Claims

The paper claims that data volume alone is insufficient to guarantee stable or reliable performance estimates for African NLI. The authors assert that the observed non-monotonic trends are systematic rather than incidental, driven by the unique linguistic and typological characteristics of the languages and the nature of the AfriXNLI benchmark.

The significance of this work lies in its call for evaluation-conscious benchmarking. The authors warn against relying on naive beliefs that "more data equals better evaluation." Instead, they propose that meaningful benchmarking in low-resource multilingual settings requires:

Reporting mean accuracy with standard deviation across multiple subsamples.
Avoiding over-reliance on small held-out test sets.
Adopting language-specific evaluation sizes (suggesting a minimum of 300 samples) to ensure stability.
Recognizing that performance drops with larger sample sizes may reflect the removal of optimistic bias rather than a failure of the model.

The study concludes that without these adjustments, current evaluations may misrepresent the true capabilities and limitations of models in African languages.