Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR
This paper proposes a continual learning framework for Automatic Speech Recognition that integrates explicit disfluency tokens to effectively handle disfluent speech on limited datasets while mitigating catastrophic forgetting and revealing a shared cross-attention mechanism across training methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very smart, well-trained robot assistant that can listen to people speak and type out exactly what they said. This robot is great at its job, but it has a habit: when people stumble, say "um," or repeat themselves, the robot just deletes those parts to make the text look clean. While this sounds helpful, it actually causes the robot to "hallucinate"—it might invent words to fill the gaps or lose important details about how the person was feeling or thinking.
The researchers in this paper wanted to teach this robot to keep those stumbling blocks (called "disfluencies") in the transcript, like a court stenographer who writes down every "uh" and "um" exactly as they happen. However, there's a catch: if you try to teach the robot this new skill using only a small amount of new data, it tends to "forget" everything it already knew about speaking clearly. This is like a student who studies so hard for a history test that they forget how to do basic math.
Here is how the paper solves this problem, explained simply:
1. The Problem: The "Clean" vs. "Real" Dilemma
Most speech robots are trained to give you a "clean" version of speech. But sometimes, you need the "real" version.
- The Goal: Teach the robot to recognize and write down specific markers for things like filler words ("um"), repetitions ("I-I-I"), coughs, and pauses.
- The Risk: If you just retrain the robot on messy, stuttering speech, it might get so confused that it stops working well on normal, clean speech. This is called "catastrophic forgetting."
2. The Solution: "Continual Learning" (The Smart Tutor)
Instead of retraining the robot from scratch, the authors used a technique called Continual Learning (CL). Think of this like a smart tutor who helps a student learn a new subject without making them forget their old ones. They tested four different "tutoring styles" (methods) to see which one worked best:
- The "Memory Buffer" (Experience Replay): The robot keeps a small notebook of old, clean conversations and reviews them while learning new, messy ones.
- The "Gentle Nudge" (EWC): The robot is told, "Don't change your brain too much on the parts you already know well."
- The "Weighted Average" (WA): The robot learns the new stuff, and then the teacher mixes the "old brain" and the "new brain" together to find a perfect balance.
- The "Gradient Guard" (A-GEM): The robot checks its new lessons against old ones to make sure it isn't taking steps backward.
3. The Discovery: The "Specialized Team"
The researchers didn't just look at how well the robot performed; they looked inside the robot's brain (specifically, its attention mechanisms) to see how it learned.
They found something fascinating: When the robot successfully learned to write down "ums" and "repetitions," it didn't use its whole brain. Instead, a small, specific team of internal processors (called attention heads) took over that job.
- The Analogy: Imagine a large office where everyone usually does everything. But when the job is "writing down stutters," a specific group of three employees steps up, sits at a special desk, and handles it exclusively.
- The Proof: When the researchers temporarily "fired" (masked) this specific team, the robot stopped writing down the stutters, but it could still understand normal speech perfectly fine. This proved that the robot had created a dedicated, stable mechanism for handling disfluencies.
4. The Results: Who Won?
The researchers tested these methods on different groups of speakers, including university students, elderly people with dementia, and people with mild cognitive impairment.
- The Trade-off: There is a tug-of-war. Methods that were very good at keeping the robot's original "clean speech" skills (like the Weighted Average method) sometimes struggled to learn the new "stutter" markers. Conversely, methods that were great at learning the markers sometimes made the robot slightly worse at clean speech.
- The Winners:
- For remembering the old stuff (Clean Speech): The Weighted Average (WA) method was the most stable. It kept the robot's original skills intact while adding the new ones.
- For remembering the new stuff (Stutters): The Experience Replay (ER) method (the one with the notebook) was the best at remembering and recognizing the different types of stumbles, especially for difficult cases like pauses.
5. The Bottom Line
This paper shows that we can teach speech robots to be more honest and detailed (verbatim) without breaking their ability to understand normal speech.
- Key Takeaway: You don't need to retrain the whole robot from scratch. By using the right "continual learning" strategy, you can add a specialized "stutter-detecting team" inside the robot's brain.
- The Insight: Successful learning isn't about changing the whole robot; it's about finding and activating a small, specific part of its brain to handle the new, messy details while leaving the rest of the brain alone.
The authors conclude that while no single method was perfect at everything, Experience Replay was best for remembering the messy details, and Weighted Average was best for keeping the robot's core skills sharp. This gives engineers a roadmap for building better, more human-like speech recognition systems.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.