🤖 machine learning

Data filtering methods for training language models

This paper presents a comparative analysis of Confident Learning and Dataset Cartography for detecting label errors in Russian text classification datasets, demonstrating that while both methods outperform random removal, their effectiveness in improving model performance depends heavily on dataset size and noise levels, with Confident Learning yielding significant gains on small, noisy datasets.

Original authors: Egor Shevchenko, Elena Bruches

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Egor Shevchenko, Elena Bruches

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human language. You give it a massive textbook filled with examples and answers. But here's the catch: some of the answers in the textbook are wrong. Maybe a sentence that should be labeled "happy" was accidentally marked "angry," or a grammatically correct sentence was marked as "wrong."

If you teach the robot using a textbook full of mistakes, it will learn those mistakes, too. This is the problem of label noise.

This paper is like a quality control inspection for three different Russian-language textbooks (datasets) used to train AI. The authors, E. Shevchenko and E. Bruches, tested two different "detective tools" to find and remove the bad examples before the robot starts studying.

Here is a simple breakdown of their experiment and what they found:

The Two Detectives

The researchers compared two automatic methods for finding errors:

The "Second Opinion" Detective (Confident Learning):
- How it works: Imagine you have a student take a test. Then, you take that same student, split them into four groups, and have each group grade the others' answers. If a student consistently gets a specific question wrong according to the other groups, but the answer key says they are right, the "Second Opinion" detective flags it as a likely error in the answer key.
- The vibe: It's thorough but aggressive. It trusts the consensus of the model's predictions.
The "Study Habits" Detective (Dataset Cartography):
- How it works: This detective watches the student study over time. If the student keeps getting a specific question right, then wrong, then right again, or just seems confused about it after many study sessions, the detective marks it as "troublesome." It looks at how the model's confidence changes over time.
- The vibe: It's more cautious. It only removes examples that the model consistently struggles to learn.

The Three Textbooks (Datasets)

They tested these detectives on three very different Russian datasets:

The Big Book (ru_emotion_e-culture): A huge collection of 49,000 forum posts about emotions. It's big and generally high-quality.
The Medium Book (RuCoLA): A collection of 8,500 sentences to check if they are grammatically correct. It's medium-sized but tricky because "correctness" can be subjective.
The Small Book (TERRa): A tiny collection of 2,300 pairs of sentences to check if one implies the other. It's small and suspected to be messy.

What Happened? (The Results)

1. The Big Book: "Don't Fix What Isn't Broken"
On the huge dataset, the textbooks were already pretty good.

The Result: When the detectives removed the "bad" examples they found, the robot actually got slightly worse at the task.
The Lesson: When you have a massive amount of data, the robot is smart enough to ignore a few bad apples on its own. Removing them just made the book smaller without making it better.

2. The Medium Book: "Better than Random, But Not Perfect"
On the medium dataset, both detectives found a lot of errors (about 15–18% of the book).

The Result: Removing these errors didn't make the robot perfect, but it did make it perform better than if you had just randomly thrown away the same number of pages.
The Lesson: The detectives were actually finding real mistakes, not just guessing. However, the dataset was still too noisy or the task too hard to see a huge jump in performance just by cleaning it.

3. The Small Book: "The Magic of Cleaning"
This was the most dramatic result. The small dataset was suspected to be very messy.

The Result: The "Second Opinion" detective found that 35% of the book was wrong! When they removed those bad examples, the robot's performance jumped significantly.
The Comparison: If they had just randomly thrown away 35% of the book, the robot would have crashed and burned. But because they threw away the specific bad examples, the robot became much smarter.
The Lesson: For small, messy datasets, finding and removing the errors is a game-changer.

The Big Takeaway

The paper concludes that there is no "one size fits all" solution.

If you have a huge, clean dataset, you don't need to waste time filtering; the AI can handle the noise.
If you have a small, messy dataset, using a tool like "Confident Learning" to clean the data is essential. It's like cleaning a small, muddy room: if you don't remove the mud, you can't see the floor.

The authors also noted that the "Study Habits" detective (Dataset Cartography) is safer and less likely to accidentally throw away good examples, while the "Second Opinion" detective (Confident Learning) is more aggressive and better at finding the worst errors in small datasets.

In short: Cleaning your data is like polishing a lens. If the lens is already clear and huge, polishing it a bit more doesn't help. But if the lens is small and covered in mud, cleaning it is the only way to see clearly.

Technical Summary: Data Filtering Methods for Training Language Models

Problem Statement

The effectiveness of supervised machine learning models is fundamentally constrained by the quality of labeled training data. Despite the shift toward "data-centric AI," label noise—discrepancies between assigned labels and true categories—remains pervasive, even in widely used benchmarks. While extensive research has addressed this in English-language contexts, systematic analysis of annotation quality in Russian-language corpora is lacking. The unique linguistic characteristics of Russian (free word order, rich morphology, homonymy) may exacerbate annotation subjectivity and error rates. This work addresses the gap by evaluating automatic label error detection methods on Russian text classification tasks to determine if targeted data filtering improves model generalization compared to random data reduction.

Methodology

The study conducts a comparative analysis of two automatic label error detection methods: Confident Learning (CL) and Dataset Cartography (DM).

Datasets

Experiments were performed on three Russian corpora varying in size, domain, and class distribution:

ru_emotion_e-culture: 49,123 examples, 5 classes (emotion classification).
RuCoLA: 8,524 examples, 2 classes (linguistic acceptability).
TERRa: 2,337 examples, 2 classes (textual entailment recognition).

Base Model and Training

All experiments utilized the pre-trained rubert-base-cased model.

Standard Fine-tuning: 3 epochs, batch size 32, learning rate $2\times10^{-5}$ .
Evaluation Metric: F1-macro (to account for class imbalance), supplemented by accuracy.

Detection Methods

Confident Learning (CL): Implemented via the cleanlab library. It uses $K$ -fold cross-validation ( $K=4$ ) to generate out-of-fold predictions. A "confident joint" matrix estimates the distribution between observed and latent true labels. Examples where the model confidently predicts a different class than the assigned label are flagged as errors.
Dataset Cartography (DM): Implemented via a custom callback. It analyzes in-sample training dynamics over 10 epochs. Four metrics are computed per example: Confidence, Variability, Correctness, and Forgetfulness. Examples are categorized into "easy," "ambiguous," and "hard-to-learn" regions based on confidence and variability thresholds. "Hard-to-learn" examples are candidates for removal.

Experimental Design

For each method, the identified "noisy" examples were removed from the training set, and the model was retrained. Crucially, control experiments were conducted where an equivalent number of examples were removed randomly. This design isolates the effect of targeted filtering from the mere reduction of training set size.

Key Contributions

First Comparative Analysis: This work provides the first comparative evaluation of Confident Learning and Dataset Cartography specifically on Russian text classification corpora.
Dependency on Dataset Characteristics: The study demonstrates that filtering effectiveness is not uniform but heavily dependent on corpus size and initial noise levels.
Validation of Filtering Meaningfulness: Through control experiments, the authors separate the benefits of targeted filtering from simple sample size reduction, proving that both methods identify substantively problematic examples.

Results

Filtering Aggressiveness

Confident Learning showed adaptive behavior, removing a small fraction of data on large corpora (3.75% on ru_emotion_e-culture) but a massive fraction on small, noisy corpora (35.47% on TERRa).
Dataset Cartography was more conservative and stable, removing between 9.28% and 14.48% of data across all corpora.

Performance Outcomes (F1-macro)

Large, High-Quality Corpus (ru_emotion_e-culture): Neither method improved performance over the baseline (F1 $\approx$ 0.935). In fact, both resulted in slight decreases, though CL performed nearly identically to random removal, suggesting the model was robust to the sparse noise present.
Medium Corpus (RuCoLA): Both methods resulted in lower absolute F1 scores than the baseline. However, both significantly outperformed random removal (CL: +0.0166 over random; DM: +0.0141 over random), confirming that the removed examples were indeed more problematic than a random subset.
Small, Noisy Corpus (TERRa): This yielded the most significant results.
- Confident Learning achieved a significant improvement in absolute quality (+0.0134 over baseline), raising F1 from 0.6444 to 0.6578.
- Crucially, CL outperformed random removal by a large margin (+0.1348), indicating that the 35% of removed examples contained the majority of the noise degrading the model.
- Dataset Cartography maintained baseline performance (F1 $\approx$ 0.6441) but did not improve it, outperforming random removal by +0.0590.

Diagnostic Insights

Dataset Cartography provided additional diagnostic metrics. The TERRa corpus exhibited significantly lower average confidence (0.811) and higher variability (0.184) compared to ru_emotion_e-culture (0.979 and 0.033, respectively), correlating with the higher noise levels and task difficulty.

Significance and Limitations

The paper concludes that targeted data filtering is a meaningful strategy, particularly for small Russian-language corpora with high annotation noise, where Confident Learning can yield substantial performance gains. For larger, cleaner datasets, the cost of filtering (data reduction) may outweigh the benefits of noise removal.

The authors explicitly note the following limitations:

Lack of Ground Truth: No manual expert verification was conducted; the "meaningfulness" of filtering is inferred indirectly via comparison with random removal rather than precision/recall against a verified error set.
Single Model/Seed: Results are based on a single base model (rubert-base-cased) and a single random seed, meaning the reported numerical differences are point estimates rather than statistically averaged results.
Scope: The analysis is restricted to text classification; generalization to other NLP tasks (e.g., sequence labeling, generation) is not addressed.

The work suggests that for small, noisy Russian datasets, applying Confident Learning followed by verification and retraining is a recommended practical approach, while Dataset Cartography serves as a safer, more computationally efficient tool for data quality diagnostics.