Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition

This paper introduces USR 2.0, a unified speech recognition framework that replaces expensive autoregressive pseudo-labelling with a novel CTC-driven teacher forcing mechanism and mixed sampling to halve training time while significantly improving robustness and achieving state-of-the-art performance across audio, visual, and audiovisual tasks.

The Gist

Imagine you are trying to teach a robot to understand speech, but you only have a tiny dictionary of correct sentences. You want it to learn from thousands of hours of videos where the audio is messy, the speakers have strange accents, or the camera is shaky. This is the challenge of Unified Speech Recognition (USR): teaching one single brain to understand speech from sound, from lip movements, or from both combined.

The previous version of this technology (called USR) worked well, but it had two big problems:

1. It was incredibly slow. It tried to read every word one by one, like a student reading a book aloud to check their work before moving to the next sentence.
2. It was fragile. If the robot made a small mistake early on, it would get confused, make more mistakes, and spiral into nonsense, especially with long or noisy sentences.

The authors of this paper (published at ICLR 2026) introduced USR 2.0, a new method that fixes these issues. Here is how it works, explained with simple analogies.

1. The Problem: The "Slow Reader" vs. The "Fast Skimmer"

To understand the solution, we need to understand the two ways the robot "reads" speech:

• The Attention Decoder (The Slow Reader): This is like a careful student reading a sentence word-by-word. It looks at the previous word to guess the next one. It's very smart and understands context well, but it's slow because it has to wait for one word before guessing the next. If it guesses wrong on word #1, it gets confused for word #2, and the whole sentence falls apart.
• The CTC Head (The Fast Skimmer): This is like a speed-reader who glances at the whole page and grabs the main keywords. It's fast and very robust (it doesn't get confused easily by noise), but it sometimes misses the fine details or the exact order of words.

The Old Way (USR): The robot tried to use the "Slow Reader" to generate its own homework answers (pseudo-labels) to teach itself. Because the "Slow Reader" is so slow, the whole training process took forever. Also, because the "Slow Reader" was prone to errors, the robot kept teaching itself the wrong things.

2. The Solution: "CTC-Driven Teacher Forcing"

The authors came up with a clever trick called CTC-driven Teacher Forcing.

Imagine a strict but efficient teacher (the "Teacher") and a student (the "Student").

• The Old Method: The teacher would read a sentence aloud word-by-word (slowly) to give the student the answer key. The student would then try to match it.
• The New Method (USR 2.0): The teacher uses their "Fast Skimmer" ability to quickly grab the main keywords and the general structure of the sentence. They hand this "skeleton" to the student.
  • The student then fills in the details using their "Slow Reader" brain, but they are forced to follow the teacher's skeleton.
  • The Magic: Because the teacher and student are both looking at the same "skeleton" at the same time, the student learns incredibly fast. Even if the teacher's skeleton isn't a perfect, grammatically beautiful sentence, it's good enough for the student to learn the pattern.

Analogy: Think of it like building a house.

• Old Way: You try to build the house brick by brick, checking every single brick's alignment before laying the next one. If you mess up the foundation, the whole wall collapses.
• New Way: You use a crane (the Fast Skimmer) to drop the main beams and walls into place instantly. Then, you (the Slow Reader) just go in and do the fine finishing work (painting, wiring) based on where the beams are. It's much faster, and the house is less likely to collapse because the heavy lifting was done by the robust crane.

3. The Safety Net: "Mixed Sampling"

There was one risk with the new method: What if the teacher's "Fast Skimmer" gave a bad skeleton? The student might learn to copy those bad habits.

To fix this, the authors added Mixed Sampling.

• Imagine a coach who switches strategies during practice.
• 50% of the time: The coach uses the "Fast Skimmer" method (CTC-driven) to keep things fast and robust.
• 50% of the time: The coach switches back to the "Slow Reader" method (standard) to make sure the student doesn't forget how to read carefully.

This keeps the student balanced: fast and tough, but also precise and detailed.

4. The Results: Why It Matters

By using this new approach, USR 2.0 achieves three major wins:

1. Speed: Training is twice as fast. The robot learns in half the time because it doesn't have to wait for the "Slow Reader" to finish every single step.
2. Robustness: The robot is much better at handling noise, long sentences, and weird accents. Because it relies on the "Fast Skimmer" (CTC) to guide the process, it doesn't get confused when the audio is messy.
3. One Model to Rule Them All: It successfully uses a single model to understand audio-only, video-only (lip reading), and audio-visual speech. It beats all previous specialized models.

Summary

In short, the authors realized that trying to be perfect and slow was holding the robot back. Instead, they taught the robot to be fast and good enough to get the general idea, then use that to learn the details. It's like learning to drive: instead of memorizing every single turn before you start the car, you learn the general flow of traffic first, then refine your driving as you go.

USR 2.0 is the result: a speech recognition system that is faster, tougher, and smarter, capable of understanding human speech in the real, messy world.

Technical Summary

1. Problem Statement

Unified Speech Recognition (USR) aims to train a single model capable of handling Audio (ASR), Visual (VSR), and Audiovisual (AVSR) speech recognition tasks simultaneously. While the original USR framework achieved state-of-the-art results using a semi-supervised approach with a teacher-student architecture, it suffers from two critical limitations:

1. Computational Bottleneck: The original method relies on autoregressive (AR) pseudo-labelling for the attention-based decoder. Generating these labels requires a slow, token-by-token forward pass at every training step, making training expensive and difficult to scale.
2. Robustness to Distribution Shifts: The original USR uses decoupled supervision, where the CTC branch and the Attention branch are trained independently. Under out-of-distribution (OOD) conditions (e.g., long sequences, noise, unseen domains), the autoregressive decoder is prone to cascading errors. These errors are reinforced in the training loop because the noisy pseudo-labels generated by the teacher degrade the student, which in turn degrades the teacher (via Exponential Moving Average updates).

2. Methodology: USR 2.0

The authors propose USR 2.0, a novel framework that combines the efficiency and robustness of Connectionist Temporal Classification (CTC) with the expressive power of attention-based decoding. The core innovations are:

A. CTC-Driven Teacher Forcing

Instead of decoding the teacher autoregressively to generate attention-based pseudo-labels, USR 2.0 uses the greedily decoded CTC outputs as input to the decoder.

• Process:
  1. The teacher's CTC head greedily decodes the input to produce a sequence of tokens (after merging blanks and repeats).
  2. These CTC tokens are fed into the teacher's attention decoder as a fixed prefix (Teacher Forcing).
  3. The decoder generates the full attention-based pseudo-label sequence in a single forward pass (parallel generation), rather than an autoregressive loop.
• Alignment: Since the attention targets are derived directly from the CTC outputs, they share the same sequence length and alignment. This allows the student decoder to predict both CTC and Attention targets simultaneously in one pass.
• Coherence Insight: While the resulting attention sequence might lack global coherence (due to conditioning on potentially imperfect CTC prefixes), this is acceptable in the pseudo-labelling setting. The student and teacher operate under the same forced inputs, ensuring effective knowledge transfer despite local inconsistencies.

B. Mixed Sampling Strategy

To address the train-test mismatch (training with CTC-derived inputs vs. inference with autoregressive inputs) and mitigate exposure bias, the authors introduce a mixed sampling strategy:

• CTC-Driven Mode (50% probability): The decoder is trained on CTC-derived pseudo-labels. This maximizes efficiency and robustness.
• Autoregressive (AR) Mode (50% probability): The teacher generates standard AR pseudo-labels, and the student is trained on them. This ensures the model retains the ability to generate coherent sequences autoregressively during inference.
• Loss Function: In CTC-driven mode, the decoder is supervised by both the teacher's attention labels and the CTC labels to leverage the strengths of both. In AR mode, the CTC branch is supervised by both label types to improve its robustness.

3. Key Contributions

1. Efficiency: By replacing autoregressive decoding with CTC-driven teacher forcing, the method eliminates the sequential dependency during pseudo-label generation. This results in ~40x faster decoding during the pseudo-labelling phase and halves the total training time.
2. Robustness: The coupling of CTC and Attention branches allows the model to inherit CTC's robustness to monotonic alignment errors and distribution shifts. This significantly reduces the "self-reinforcing error" loop common in semi-supervised learning.
3. Unified Architecture: USR 2.0 maintains a single model for ASR, VSR, and AVSR, achieving state-of-the-art performance across all modalities without needing task-specific fine-tuning or separate models.
4. Scalability: The efficiency gains allow for scaling to larger models (up to a "Huge" model with 953M parameters) and larger unlabelled datasets (2,500 hours).

4. Experimental Results

The paper evaluates USR 2.0 on several benchmarks, including LRS3, LRS2, WildVSR, and AVSpeech.

• Out-of-Distribution (OOD) Robustness:
  • Long Sequences: USR 2.0 maintains low Word Error Rates (WER) on long utterances (up to 600 frames), whereas previous methods (USR, AV-HuBERT) suffer sharp performance degradation due to autoregressive drift.
  • Noise: On noisy audio (SNR -5dB), USR 2.0 outperforms baselines, particularly in AVSR where visual cues help, but the robust decoder also aids ASR.
  • Greedy Decoding: USR 2.0 performs exceptionally well with greedy decoding (no beam search), making it ideal for low-latency applications and efficient pseudo-labelling.
• In-Distribution Performance:
  • USR 2.0 achieves State-of-the-Art (SOTA) results on LRS3, LRS2, and WildVSR.
  • Huge Model Results: The scaled-up model achieves WERs of 0.8% (AVSR), 0.9% (ASR), and 17.6% (VSR) on LRS3, surpassing methods that use separate models or massive external language models.
• Training Efficiency:
  • USR 2.0 converges faster, requiring fewer epochs (50 vs. 75 for USR) to reach comparable or better performance.
  • Total training time is reduced by nearly 50%.

5. Significance and Impact

• Paradigm Shift in Semi-Supervised Learning: The paper demonstrates that global sequence coherence is not strictly necessary for effective pseudo-labelling if the teacher and student share the same conditioning. This insight allows for the use of fast, non-autoregressive generation techniques in self-training loops.
• Practical Deployment: By reducing training costs and improving robustness to real-world noise and domain shifts, USR 2.0 makes unified speech recognition more viable for deployment in diverse, uncontrolled environments.
• Broader Applicability: The "CTC-driven teacher forcing" paradigm is suggested as a generalizable technique for other sequence-to-sequence tasks (e.g., handwriting recognition, DNA sequencing) where input and output share temporal order but lack explicit frame-level alignment.

In summary, USR 2.0 successfully resolves the trade-off between the speed of CTC and the expressiveness of attention mechanisms, creating a highly efficient and robust unified speech recognition framework.