💬 NLP

Corpora deduplication or duplication in Natural Language Processing of few resourced languages ? A case of study: The Mexico's Nahuatl

This paper investigates whether controlled data duplication can effectively enhance Natural Language Processing for low-resource languages, demonstrating that an incremental duplication technique applied to the Nahuatl corpus yields moderate improvements in sentence-level semantic similarity tasks.

Original authors: Juan-José Guzman-Landa, Juan-Manuel Torres-Moreno, Graham Ranger, Miguel Figueroa-Saavedra, Martha-Lorena Avendaño-Garrido, Elvys Linhares-Pontes, Luis-Gil Moreno-Jiménez

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juan-José Guzman-Landa, Juan-Manuel Torres-Moreno, Graham Ranger, Miguel Figueroa-Saavedra, Martha-Lorena Avendaño-Garrido, Elvys Linhares-Pontes, Luis-Gil Moreno-Jiménez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Empty Library"

Imagine you want to teach a robot to speak Nahuatl, a beautiful indigenous language spoken in Mexico by over a million people. To teach a modern AI (like a Large Language Model), you usually need a massive library of books—billions of words—to show the robot how the language works.

But for Nahuatl, the library is almost empty. There are very few digital books, and the language is incredibly complex. It's an agglutinative language, which means speakers glue many small pieces of meaning together to form one giant word.

Analogy: In English, you might say, "I am going to the store." In Nahuatl, that whole sentence might be packed into a single, super-long word. Because of this, and because there are so few digital texts, the AI has very little to learn from. It's like trying to teach someone to cook by giving them only three ingredients.

The Controversial Idea: "Copy-Paste" Cooking

Usually, when computer scientists build AI, they are obsessed with deduplication. They spend hours deleting duplicate sentences because they believe repeating the same text makes the AI "lazy" or confused. They think, "If the AI sees the same sentence 100 times, it won't learn anything new."

This paper asks a crazy question: What if, for languages with almost no data, repeating the text is actually a good thing?

The researchers decided to treat the small Nahuatl library like a photocopier. Instead of throwing away duplicates, they took their tiny collection of Nahuatl texts and photocopied them over and over again.

They took the original text (let's call it 1x).
They made 2 copies (2x).
They made 4 copies (4x).
They went all the way up to 30 copies of the same text.

They wanted to see if feeding the AI the same story 30 times would help it understand the language better than just reading it once.

The Experiment: Training the AI

They used this "stretched" library to train three different types of AI models (think of them as different student learning styles):

FastText (The detailed learner who looks at small parts of words).
Word2Vec (The pattern finder).
GloVe (The statistical calculator).

They tested these models on a task called Semantic Similarity.

The Test: They gave the AI a sentence and asked it to pick the most similar sentence from a list of five options.
The Goal: Did the AI get better at understanding the meaning of the words after seeing the text repeated 30 times?

The Results: The "Goldilocks" Zone

The results were surprising and showed that "more repetition" isn't always "more confusion."

The Winners: The FastText and Word2Vec models got significantly better at understanding Nahuatl when they saw the text repeated.
- FastText peaked when the text was copied about 10 times.
- Word2Vec peaked when the text was copied about 22 times.
- Analogy: Imagine a student trying to learn a difficult song. Reading the sheet music once isn't enough. Reading it 10 times helps them get the rhythm. Reading it 20 times helps them master the notes. But reading it 100 times might just make them bored and tune out. The researchers found the "sweet spot" where the repetition helped the most.
The Loser: The GloVe model didn't like the repetition at all; its performance actually got worse.
The Comparison: The AI trained on this "repeated" Nahuatl text performed better than AI models trained on other available Nahuatl data (like the Nahuatl Wikipedia). Even though the data was just copies of the same thing, the sheer volume helped the AI learn the structure of the language.

Why Does This Matter?

This paper suggests a new rule for "resource-poor" languages (languages with very little digital data):

For rich languages (like English or Spanish): We should delete duplicates to keep the data clean and diverse.
For poor languages (like Nahuatl): We might need to duplicate our limited data to give the AI enough "practice" to learn the rules.

The Conclusion

The researchers found that by carefully copying their small Nahuatl dataset up to 30 times, they could teach an AI to understand the language's meaning much better than before. It's a bit like practicing a speech in front of a mirror: if you only have one mirror, you might not see all your mistakes. But if you stand in a room with 30 mirrors, you can see every angle and perfect your performance.

In short: When you have very little data, sometimes the best way to learn is to repeat, repeat, repeat—until you find the perfect amount of repetition.

1. Problem Statement

The paper addresses the critical challenge of developing Natural Language Processing (NLP) tools for $\pi$ -languages (languages with few computational resources), specifically Nawatl (Nahuatl), an agglutinative and polysynthetic indigenous language of Mexico.

Data Scarcity: Large Language Models (LLMs) and contextual embedding models (like BERT) require massive datasets (hundreds of millions to billions of tokens) to function effectively. $\pi$ -languages suffer from a severe lack of representative, large-scale textual corpora.
Limitations of Existing Solutions: Standard data augmentation techniques (e.g., EDA, synonym substitution) rely on linguistic resources like dictionaries or Part-of-Speech (POS) taggers, which are largely non-existent or insufficient for Nawatl due to its complex morphology and dialectal diversity.
The Core Question: Can controlled data duplication (intentionally repeating the same text) serve as a viable strategy to expand corpora and improve the training of static word embeddings for low-resource languages, contrary to the standard practice of deduplication used for high-resource languages?

2. Methodology

The authors propose an Incremental Duplication technique applied to the $\pi$ -YALLI corpus, a heterogeneous collection of Nawatl texts containing approximately 6.6 million words across various dialects.

Experimental Design:
- The original corpus was duplicated $\rho$ times, where $\rho$ ranges from $1\times$ (original) to $30\times$ (30 copies), creating datasets ranging from ~6.6 million to ~198 million words.
- The duplication was "identical" (exact copies) to avoid introducing syntactic or semantic biases associated with random augmentation.
Models Trained:
- Three static embedding algorithms were trained on the expanded corpora: FastText, Word2Vec, and GloVe.
- Both Skip-gram and CBOW architectures were tested for FastText and Word2Vec.
Evaluation Task:
- Sentence Semantic Similarity: The quality of the embeddings was evaluated by ranking candidate sentences based on their semantic proximity to reference sentences.
- Metric: Performance was measured using Kendall's $\tau$ coefficient, which calculates the correlation between the model's ranking and the ground truth rankings provided by native Nawatl speakers.
Baselines: Results were compared against pre-trained models (FastText/Word2Vec) trained on the Nawatl Wikipedia and the Common Crawl corpus.

3. Key Contributions

Paradigm Shift: The paper challenges the prevailing NLP dogma that data deduplication is always superior. It argues that for $\pi$ -languages, controlled duplication is a necessary and effective strategy to compensate for data scarcity.
Resource-Free Augmentation: The proposed method requires no external linguistic resources (dictionaries, POS taggers), making it uniquely suitable for agglutinative languages like Nawatl where such resources are scarce.
Empirical Validation: It provides the first empirical evidence that increasing corpus size via exact duplication improves the performance of static embeddings in low-resource scenarios.

4. Results

The experiments yielded significant insights regarding model performance relative to corpus size:

Overall Improvement: Incremental duplication led to moderate to significant improvements in semantic similarity tasks compared to the original corpus.
Model-Specific Performance:
- FastText (Skip-gram): Achieved the best overall performance. It reached a peak Kendall's $\tau$ of 0.495 (up from 0.459) with a 7.8% improvement at a duplication ratio of $\rho=10$ .
- Word2Vec (Skip-gram): Showed the most dramatic relative gain, improving from $\tau=0.357$ to 0.483 (a 35.3% increase) at $\rho=22$ .
- GloVe: Performed poorly and stagnated as duplication increased, showing only a marginal 3.4% improvement.
- CBOW Architectures: Generally underperformed compared to Skip-gram variants.
Comparison to Baselines: The models trained on the duplicated $\pi$ -YALLI corpus significantly outperformed pre-trained models based on the Nawatl Wikipedia and Common Crawl, demonstrating that the specific domain and dialectal coverage of the duplicated corpus were more valuable than larger, generic datasets.
Optimal Ratio: The study identified an "optimal" duplication point (around 8–10x for FastText and 20–22x for Word2Vec) beyond which performance gains plateaued or slightly declined, suggesting diminishing returns.

5. Significance and Future Work

Significance: This research offers a practical, low-cost solution for the NLP community to develop tools for endangered and low-resource languages. It proves that for languages with limited data, quantity (via duplication) can compensate for the lack of diversity in training static embeddings, enabling better performance in tasks like semantic similarity.
Implications: The findings suggest that the "deduplication" rule for high-resource languages should not be blindly applied to low-resource contexts.
Future Directions: The authors plan to:
- Investigate text normalization techniques to further enhance duplication efficacy.
- Apply these expanded corpora to more complex tasks such as Automatic Text Summarization and Named Entity Recognition (NER) (specifically for place names).
- Explore targeted duplication based on specific dialects or topics.

In conclusion, the paper demonstrates that for agglutinative, low-resource languages like Nawatl, controlled corpus duplication is a viable and effective technique to enhance the learning of static word embeddings, bridging the gap between data scarcity and the requirements of modern NLP models.