Technical Summary: CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition

1. Problem Statement

Modern deep learning models for Automatic Speech Recognition (ASR) often achieve high aggregate performance but consistently underperform on specific subpopulations, particularly in multilingual settings. While Group Distributionally Robust Optimization (Group DRO) has been proposed to mitigate these disparities by minimizing the worst-group loss, it fails in ASR contexts when group losses are not directly comparable.

The core issue lies in the Connectionist Temporal Classification (CTC) loss function, which is standard for non-autoregressive ASR models (e.g., XLS-R, MMS). The CTC loss scales with input sequence length ($D$) and output sequence length ($U$). Because the number of valid alignments increases with sequence length, longer audio samples inherently yield higher CTC losses, even if the transcription error rate is low. Consequently, in multilingual datasets where languages vary significantly in utterance length and acoustic properties, the raw CTC loss does not accurately reflect model performance. Group DRO, which relies on these raw losses to up-weight difficult groups, mistakenly assigns excessive weight to languages with long utterances (high loss due to length) rather than those with genuine performance deficits, leading to degraded overall and worst-case performance.

2. Methodology: CTC-DRO

The authors propose CTC-DRO, a training algorithm designed to adapt Group DRO for the specific scaling properties of CTC loss. The method introduces two primary mechanisms:

A. Length-Matched Group Losses

To address the incomparability of losses caused by varying audio durations, CTC-DRO employs a specialized batch sampler. Instead of standard random sampling, the sampler constructs batches where the total audio duration is approximately constant across groups.

• Mechanism: For a randomly selected group (language), the sampler iteratively adds utterances until a fixed duration target (hyperparameter) is met.
• Loss Aggregation: Within a batch, the utterance-level CTC losses are summed rather than averaged. This sum, representing the total loss for a fixed duration of audio, serves as the group loss ($L_g$) for weight updates. This mitigates the bias where longer utterances naturally accumulate higher losses.

B. Smoothed Maximization Objective

Standard Group DRO updates group weights ($q_g$) using an exponential update rule based on the loss $L_g$. This can lead to "over-emphasis," where a group with consistently high losses (due to irreducible factors or length) receives a disproportionately large weight, starving other groups of training signal.

• Mechanism: CTC-DRO introduces a smoothing hyperparameter $\alpha$ into the weight update rule:
  $$q_g \leftarrow \frac{q_g \cdot \exp\left(\eta_q \frac{L_g}{q_g + \alpha}\right)}{\sum_{g' \in G} q_{g'} \cdot \exp\left(\eta_q \frac{L_{g'}}{q_{g'} + \alpha}\right)}$$
• Effect: As $\alpha$ increases, the update becomes less sensitive to the current weight $q_g$ relative to the loss, preventing any single group from dominating the weight distribution. The authors prove that this update rule optimizes a generalized objective: $\max_q \sum \log(q_g + \alpha)L_g$, which maintains the property of up-weighting high-loss groups while ensuring a more balanced distribution of weights.

3. Key Contributions

1. Identification of CTC Incompatibility: The paper demonstrates that standard Group DRO is fundamentally incompatible with CTC-based ASR training due to the length-dependent scaling of the loss function, which creates spurious differences between group losses.
2. CTC-DRO Algorithm: A novel optimization algorithm that combines length-matched batching with a smoothed weight update rule to stabilize group weights and prevent over-emphasis on specific groups.
3. Empirical Validation: Extensive evaluation on the ML-SUPERB 2.0 benchmark across five diverse language sets, utilizing pre-trained models XLS-R and MMS.

4. Experimental Results

The authors evaluated CTC-DRO against standard baselines and standard Group DRO models on balanced and unbalanced data settings.

• Worst-Case Performance: CTC-DRO consistently reduced the Character Error Rate (CER) of the worst-performing language across all five language sets.
  • Maximum Improvement: A relative reduction of 47.1% in the worst-language CER was observed (Set 2, XLS-R with additional data).
  • Average Improvement: Across all sets, the worst-language CER was reduced by up to 47.1%, with average error reductions of up to 32.9%.
• Comparison with Baselines:
  • Vs. Standard Group DRO: Standard Group DRO frequently worsened performance, increasing the worst-language CER by up to 57.5% in some settings and increasing average CER in all settings.
  • Vs. Baseline (CTC only): CTC-DRO improved both worst-case and average performance without significantly degrading the performance of the best-performing languages.
• Ablation Study: Removing either the length-matched batching or the smoothed maximization objective resulted in substantial performance degradation, confirming that both components are necessary. Specifically, removing the smoothing objective caused group weights to fluctuate wildly and concentrate on single languages, mimicking the failure of standard Group DRO.
• Scalability: The method remained effective when scaled to 18 languages, reducing the worst-language CER by 8.9% (MMS) and 9.2% (XLS-R) in balanced settings.

5. Significance and Claims

The paper claims that CTC-DRO offers a practical, low-cost solution to reduce group disparities in multilingual ASR.

• Minimal Computational Overhead: The algorithm requires only tracking a single scalar weight per group and uses standard gradient descent/ascent steps, incurring negligible computational costs compared to baseline training.
• Generalizability: While motivated by multilingual ASR, the authors suggest the principles of smoothed maximization and length-matched batching could apply to other domains involving variable-length sequences and incomparable group losses, such as medical applications, text classification, and video transcription.
• Limitations: The authors modestly note that while disparities are reduced, they are not eliminated. The approach relies on pre-defined groups (languages), and future work could explore automatic grouping or application to pre-training phases.

In summary, CTC-DRO addresses a specific failure mode of Group DRO in speech recognition, enabling more equitable performance across diverse languages without the prohibitive computational costs of alternative robust optimization methods.