💬 NLP

MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition

This paper introduces MoDiCoL, a modular diagnostic continual learning dataset and curriculum designed to address the limitations of existing benchmarks by simulating real-world distribution shifts and evaluating how speech recognition models acquire, transfer, and retain robustness under evolving acoustic and linguistic conditions.

Original authors: Theresa Pekarek Rosin, Matthias Kerzel, Stefan Wermter

Published 2026-06-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Theresa Pekarek Rosin, Matthias Kerzel, Stefan Wermter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've trained a very smart robot to listen to people speak and write down what they say. This robot is a master at its job when everything is perfect: the speaker is a calm adult with a standard accent, speaking clearly in a quiet room. But in the real world, things are rarely perfect. People might have a heavy accent, speak while coughing, talk over a noisy fan, or be a child with a high-pitched voice. When the robot encounters these messy real-world situations, it often gets confused and makes mistakes.

The problem is that most tests for these robots only check one thing at a time. They might test how the robot handles noise, or how it handles accents, but never both at once. It's like testing a car's brakes on a dry track, then its steering on a wet track, but never seeing how it handles a wet track while you're slamming on the brakes.

The authors of this paper, MoDiCoL, decided to build a better testing ground and a new way to train these robots. Here is how they did it, using some simple analogies:

1. The "Recipe Book" of Speech (The Dataset)

The researchers created a massive library of speech called MoDiCoL. Instead of just grabbing random recordings from the internet, they built this library like a chef building a menu with a strict recipe.

They identified three main ingredients that change how speech sounds:

The Script (Linguistic Content): What is being said? Is it medical jargon, air traffic control commands, or just casual chat?
The Voice (Speaker Characteristics): Who is saying it? Is it a child, an elderly person, someone with a speech impairment, or someone with a specific accent?
The Room (Acoustic Environment): Where is it being said? Is it quiet, is there a fan buzzing, or is there a crowd of people talking (babble noise)?

They mixed and matched these ingredients in a scientific way. Since they couldn't find real recordings for every possible combination (like an elderly person with a specific accent speaking medical terms in a noisy room), they used AI voice generators to create the missing pieces. Think of it like using a high-tech 3D printer to build the exact missing puzzle piece you need to complete the picture.

2. The "Gym Workout" for the Robot (Continual Learning)

Usually, you train a robot once on a huge pile of data and then leave it alone. But in the real world, the robot needs to keep learning as new situations arise.

The authors designed a curriculum (a training schedule) that acts like a progressive gym workout for the robot:

Level 1 (The Warm-up): The robot learns to handle noisy rooms.
Level 2 (The Weights): The robot learns to understand different types of voices (children, elderly, impaired).
Level 3 (The Cardio): The robot learns to understand complex topics and casual conversation styles.
Level 4 (The Marathon): The robot faces the hardest challenge: a noisy room, a difficult voice, and complex speech all at once.

The goal was to see if the robot could learn these new skills without forgetting the old ones. This is the "Continual Learning" part. It's like a student learning French, then Italian, then Spanish, without forgetting how to speak French.

3. The Three Training Coaches (The Methods)

To see how well the robot could learn without forgetting, they tried three different "coaches" (algorithms):

Coach 1: The Note-Taker (Experience Replay). This coach keeps a small notebook of old examples. When the robot learns something new, the coach says, "Wait, let's review a few old notes first." This helps the robot remember the past while learning the present.
Coach 2: The Statue Keeper (Representation Regularization). This coach tries to freeze the robot's "brain structure" so it doesn't change too much. It's like trying to keep a statue perfectly still while someone tries to paint new details on it.
Coach 3: The Traffic Controller (Orthogonal Gradient Descent). This coach makes sure the robot's new learning moves in a completely different direction than its old learning, so the two don't crash into each other. It's like telling a car, "You can turn left to learn this new thing, but don't turn right, because that's where your old knowledge lives."

4. What They Found

The results were quite revealing:

The Robot was Fragile: Before this special training, the robot did great in quiet, standard conditions but fell apart when faced with accents, impairments, or noise.
The Note-Taker Won: The "Note-Taker" coach (Experience Replay) was the best. By keeping a small memory of past examples (about 10% of the data), the robot learned new skills without forgetting the old ones. In fact, it performed even better than if they had trained it on all the data at once!
The "Statue" Failed: The coach that tried to freeze the robot's brain (Representation Regularization) caused the robot to forget a lot. It seems the robot needed to be flexible, not rigid.
The Order Matters: It mattered which skill the robot learned first. If the robot learned the hardest things first, it struggled later. But if it learned step-by-step, it got better at adapting.
Surprise Finding: When the robot faced the "Marathon" level (all difficulties at once), it didn't necessarily perform the worst. Sometimes, learning the complex mix actually helped it recover performance on the simpler tasks.

The Bottom Line

The paper introduces a new, highly organized way to test and train speech recognition systems. They showed that if you train these systems gradually, like a student taking classes one by one, and give them a small memory of what they've learned before, they become much more robust. They don't just get better at hearing; they become better at remembering how to hear in a messy, changing world.

The researchers have made their "recipe book" (the dataset) and their "training schedule" available for others to use, so more robots can learn to handle the real world without forgetting how to speak.

Technical Summary: MoDiCoL – A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition

1. Problem Statement

Modern Automatic Speech Recognition (ASR) systems have achieved high performance on standard benchmarks but struggle significantly under real-world distribution shifts. These shifts arise from variations in recording conditions, accents, speech impairments, and noise. Existing benchmarks typically isolate these factors for separate evaluation, failing to reflect the cumulative nature of distribution shifts in real-world applications. Furthermore, while Continual Learning (CL) has been applied to domain adaptation (e.g., accent or language), its potential as a diagnostic tool to identify where and why forgetting occurs in pretrained ASR models remains underexplored compared to Vision-Language models. Current datasets often lack the controlled, factorial design necessary to rigorously analyze the interplay between linguistic content, speaker characteristics, and acoustic environments during incremental learning.

2. Methodology

2.1. The MoDiCoL Dataset

The authors introduce MoDiCoL (Modular Diagnostic Continual Learning), a dataset designed for the controlled analysis of ASR robustness. It employs a systematic factorial design to evaluate three primary factor categories:

Linguistic Content: Vocabulary domain (Medical, Air Traffic Control, Other) and speech style (Read, Spontaneous, Conversational).
Speaker Characteristics: Age (Child, Adult, Elderly), Accent (English, South-East Asian, Other), Health (Healthy, Impaired), and the presence of pauses and disfluencies.
Acoustic Environment: Noise type (Clean, Babble, Fan), Signal-to-Noise Ratio (SNR) levels, and microphone distance.

Construction and Composition:
The dataset utilizes an $L_{27}$ orthogonal array (Taguchi design) combined with two foldover dimensions to generate 108 distinct run configurations. Each configuration contains 75 samples, resulting in a total of 8,100 samples (18.79 hours of speech).

Data Sources: The dataset combines real-world open-source data (e.g., TED-LIUMv2, Common Voice, NNCES) with synthetic speech generated via XTTS-v2.
Handling Constraints: Combinations unlikely to occur in reality (e.g., medical speech from children) are synthesized using voice cloning. Logically infeasible combinations (e.g., specific noise types with clean SNR) are repaired by adjusting the noise type.
Augmentation Pipeline: A controlled pipeline applies denoising, disfluency insertion, impairment simulation (via prosodic/spectral modifications), pause manipulation, distance simulation (reverberation), and noise injection to ensure samples match specific factor levels.

2.2. Continual Learning Curriculum

The authors propose a real-world-inspired CL curriculum to simulate incremental updates. The curriculum consists of a control task ( $t_0$ ) followed by four drift tasks:

$t_1$ (Acoustic Drift): Introduction of noise, varied SNR, and distance.
$t_2$ (Speaker Drift): Introduction of new speaker groups (children, elderly, impaired) and accents.
$t_3$ (Linguistic Drift): Exposure to new domains (medical, ATC) and styles (spontaneous, conversational).
$t_4$ (Compound Drift): A combination of multiple drift types (e.g., impaired speakers in noisy, distant environments).

The study evaluates the model under different permutations of this curriculum to analyze robustness transfer and the impact of task ordering.

2.3. Experimental Setup

Base Model: Pretrained whisper-small.en.
CL Strategies: Three methods are evaluated:
1. Experience Replay (ER): Maintains a memory buffer (5% and 10% sizes) of previous tasks to interleave with current training data.
2. Representation-level Regularization (RLR): Constrains the encoder to preserve previously learned representations by minimizing cosine similarity loss against frozen encoder copies.
3. Orthogonal Gradient Descent (OGD): Projects parameter updates onto the orthogonal complement of stored gradient directions to prevent interference.
Evaluation Metrics: Average Word Error Rate (A-WER), Average Incremental WER (AI-WER), Forgetting Measure (FM), Backward Transfer (BWT), Forward Transfer (FWT), and Intransigence Measure (IM).
Setting: Online, streaming CL (batch size of 1, single pass per sample) to approximate real-world adaptation.

3. Key Results

3.1. Baseline Performance

Before CL training, the unadapted whisper-small.en model performed well on the control setting ( $t_0$ ) but degraded significantly under distribution shifts.

Drift Sensitivity: Performance dropped moderately for acoustic drift ( $t_1$ ) but severely for speaker ( $t_2$ ) and linguistic ( $t_3$ ) drifts, with WER increasing dramatically.
Compound Drift: Interestingly, the compound drift ( $t_4$ ) did not yield the worst performance, suggesting that drift effects do not necessarily accumulate linearly in difficulty.
Synthetic vs. Real: Synthetic speech generally performed better than real-world speech, likely due to the higher variability and complexity of the real-world subset.

3.2. Continual Learning Performance

Experience Replay (ER): The most effective strategy. ER with a 10% buffer size achieved the lowest A-WER (17.31 ± 0.48), outperforming both other CL methods and the joint training baseline. It demonstrated minimal forgetting (FM close to 0) and strong retention of robustness.
Orthogonal Gradient Descent (OGD): Outperformed RLR across most metrics and achieved the best AI-WER (21.19 ± 1.18). The near-zero cosine similarity between stored task gradients indicated that task gradients lie in distinct subspaces, suggesting that forgetting in other methods arises from destructive updates to these gradients rather than representational drift.
Representation-level Regularization (RLR): Exhibited significant forgetting (FM: -22.34) and performed worse than OGD, potentially because mean pooling enforces global similarity, missing performance-critical local structures.
Task Order: While sequential updates generally improved learning plasticity (indicated by high IM scores), the stability of the model (FM and BWT) was highly dependent on the specific task order, with ER-10% showing the most robustness to permutation changes.

4. Significance and Claims

The paper claims that MoDiCoL serves as a diagnostic tool to investigate forgetting mechanisms in pretrained ASR models. By treating robustness as a dynamic capability that evolves across sequential distribution shifts, the authors demonstrate that:

CL as a Diagnostic: Continual learning can reveal whether performance loss is due to representational drift or destructive gradient interference. The results suggest that for ASR, interference within gradient subspaces is a primary contributor to forgetting.
Replay Efficacy: Moderate-sized memory buffers (10% replay) are highly effective at preserving robustness across diverse drift types, even outperforming joint training.
Modular Analysis: The factorial design allows for the isolation of specific factors (linguistic, speaker, acoustic) to understand their individual and combined impacts on model adaptation.

The authors conclude that CL not only preserves model robustness but also provides a framework for analyzing how robustness is acquired, transferred, and forgotten. The dataset, augmentation pipeline, and curriculum are made publicly available to facilitate further research into model behavior under evolving conditions.