⚡ electrical engineering

Improving Large-Scale Weakly Supervised ASR by Filtering and Selection

This paper proposes a novel three-stage training approach for large-scale weakly supervised ASR that combines initial pretraining with character error rate-based filtering and domain-specific acoustic selection, successfully reducing character error rates by up to 6.4% on a 90,000-hour Japanese dataset.

Original authors: Kohei Matsuura, Masato Mimura

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kohei Matsuura, Masato Mimura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human speech. The best way to do this is to feed it millions of hours of conversations. But here's the catch: you don't have perfect transcripts (the written text matching the audio). Instead, you have a massive, messy pile of audio and text scraped from the internet. Some of the text is perfect, but a lot of it is full of typos, missing words, or completely wrong because the computer that generated it made mistakes. This is what researchers call a "weakly supervised" dataset.

The paper by Matsuura and Mimura is like a recipe for turning that messy pile of data into a super-smart speech-recognition teacher, using a three-step process of Filtering and Selection.

Here is how they did it, explained simply:

The Problem: The "Noisy Classroom"

Imagine a classroom with 90,000 hours of students speaking. The teacher (the AI) is trying to learn what they are saying. However, the teacher's textbook (the labels) is full of errors. Sometimes the textbook says "apple" when the student said "pear." Sometimes it adds words that weren't spoken at all. If the teacher tries to memorize this messy textbook, they will get confused and perform poorly.

The Solution: A Three-Step Training Camp

The authors propose a clever way to clean up the learning process without throwing away the valuable data.

Step 1: The "Rough Draft" (Pretraining)

First, they let the AI study the entire messy dataset, errors and all.

The Analogy: Think of this as a student taking a quick, rough look at a whole library of books, even the ones with typos. They aren't trying to be perfect yet; they are just getting a general sense of how language works.
The Result: The AI learns the basics of speech, but it's still a bit confused by the bad labels.

Step 2: The "Quality Control" (Filtering)

Now comes the magic trick. The AI takes the messy textbook it just studied and tries to "read" the audio back to itself. It compares what it heard against what the original messy label said.

The Analogy: Imagine the student reads a sentence from the messy textbook out loud. If the text says "The cat sat on the mat" but the student hears "The cat sat on the hat," the student knows something is wrong.
The Action: They calculate a "Confusion Score" (called Character Error Rate or CER). If the score is too high, it means the label is probably garbage (too many typos or missing words). They throw those specific bad examples away.
The Surprise: They didn't just throw away the bad stuff; they kept the "okay" stuff and the "good" stuff. Then, they made the AI study this cleaned-up pile again.
The Result: Even though they were re-studying data the AI had already seen, removing the "noise" made the AI significantly smarter. It's like studying a textbook where someone finally fixed all the typos.

Step 3: The "Specialized Tutor" (Selection)

Finally, they wanted to see if the AI could get really good at a specific type of speech (like a noisy coffee shop or a formal lecture hall).

The Analogy: Imagine the AI is a generalist who knows how to talk to everyone. Now, you want it to be an expert at understanding people in a loud cafe. Instead of finding new data, they look at the pile they just cleaned up and ask: "Which of these 90,000 hours of audio sounds most like a loud cafe?"
The Action: They use a mathematical "similarity score" to pick the top 500,000 samples that sound like the target environment. They then give the AI a short, intense training session (fine-tuning) using only these similar samples.
The Result: The AI became a specialist. It didn't need a new dataset; it just needed to focus on the right part of the old dataset.

The Big Takeaway

The paper tested this on a massive Japanese dataset (90,000 hours).

Filtering (Step 2) improved the AI's accuracy by about 6.4%.
Selection (Step 3) improved it by another 4.0%.
Combined, they reduced errors by nearly 9% just by reusing and refining the same dataset.

The Key Lesson: You don't always need more data. Sometimes, you just need to be smarter about which data you use. By filtering out the "garbage" labels and then picking the "most relevant" samples, you can turn a messy, weakly supervised dataset into a powerful training tool. It's the difference between studying a messy, error-filled notebook and studying a curated, high-quality summary of the same information.

Technical Summary: Improving Large-Scale Weakly Supervised ASR by Filtering and Selection

Problem Statement
Large-scale weakly supervised datasets are essential for training robust end-to-end automatic speech recognition (ASR) models. However, these datasets, often sourced from the Internet and internal archives, suffer from two primary limitations: (1) the presence of noisy labels, including misspellings, unspoken supplementary text, and timestamp misalignments; and (2) a lack of domain specificity, which can leave models sub-optimal for specific target domains. While semi-supervised learning often employs confidence-based filtering, the effectiveness of such strategies on large-scale weakly supervised datasets remains under-explored. Furthermore, existing data selection research primarily focuses on core-set selection for reducing training time rather than improving robustness through domain adaptation using the same dataset.

Methodology
The authors propose a novel three-step training approach designed to iteratively refine a model using a single large-scale weakly supervised dataset.

Step 1: Initial Pretraining: An encoder-only ASR model based on Connectionist Temporal Classification (CTC) is pre-trained on the entire large-scale dataset. The authors select a CTC-based model for its robustness to hallucinations and its efficiency in generating predictions for the subsequent filtering step. This step allows the model to coarsely learn the ASR task despite the noise, as the majority of labels reflect the actual speech content.
Step 2: Filtering and Continued Pretraining: The model from Step 1 is used to transcribe all speech in the dataset. The Character Error Rate (CER) is calculated between the model's predictions and the original noisy labels. Samples with a CER exceeding a predefined threshold $r$ are filtered out. The model then undergoes continued pretraining on this filtered subset. The authors hypothesize that while low thresholds ensure label quality, they risk removing "hard" but correctly labeled samples, potentially biasing the model toward easier data. Thus, an optimal threshold balances label quality with data diversity.
Step 3: Data Selection and Fine-tuning: If a target-domain training set is unavailable, the authors propose selecting a subset of samples from the filtered dataset that are acoustically similar to the target domain. This is achieved by computing the cosine similarity between the mean self-supervised learning (SSL) embedding of the target domain and the embeddings of samples in the weakly supervised dataset. The top $N$ most similar samples are selected to fine-tune the continually pretrained model. This method leverages expressive SSL embeddings while maintaining linear complexity.

Key Contributions

Novel Training Pipeline: The paper introduces a framework that integrates data filtering and selection into a continuous training loop, reusing samples from the initial pretraining phase to further improve model performance.
CER-Based Filtering Analysis: The study empirically validates the impact of CER-based filtering on a 90,000-hour Japanese dataset, identifying a trade-off between label quality and data diversity.
Domain Adaptation via Similarity: The authors demonstrate that fine-tuning with acoustically similar samples selected from the same dataset used for pretraining effectively adapts the model to target domains, even without additional external data.

Experimental Results
Experiments were conducted on a 90,000-hour Japanese weakly supervised dataset and evaluated on three public datasets: CSJ (academic lectures), CommonVoice (crowdsourced), and Noisy-KU (noisy spontaneous speech).

Filtering Efficacy: In Step 2, filtering based on CER thresholds resulted in consistent improvements across all datasets. The optimal threshold was found to be between 20 and 40. This range yielded a relative Character Error Rate (CER) reduction of up to 6.4%. The authors noted that extremely low thresholds (e.g., $r \le 10$ ) provided limited gains, reinforcing the importance of maintaining data diversity.
Selection Efficacy: In Step 3, fine-tuning on the Noisy-KU dataset using acoustically similar samples (selected with $r=30$ ) achieved a relative CER reduction of 4.0% compared to fine-tuning with randomly selected samples.
Overall Performance: The proposed approach reduced the CER on the Noisy-KU dataset from 38.8% to 35.3% (a 9.0% relative reduction) solely by narrowing and reusing the single training dataset. The proposed CTC-based model also outperformed the Whisper Small model (which has twice the parameter count) on Japanese benchmarks, even without specialized Japanese pretraining data for Whisper.

Significance and Claims
The paper claims that its approach effectively exploits large-scale weakly supervised datasets by systematically addressing label noise and domain mismatch. The authors emphasize that their method is practical because it does not require additional external datasets for filtering or selection; instead, it refines the model using the existing data. The study highlights that the benefits of filtering and selection persist even after fine-tuning on target-domain data, suggesting the approach is widely applicable. To the authors' knowledge, this is the first study to demonstrate further improvement of a robust pretrained ASR model by selecting an appropriate subset from the same dataset used for pretraining.