Convex Low-resource Accent-Robust Language Detection in Speech Recognition

Technical Summary: Convex Low-resource Accent-Robust Language Detection in Speech Recognition

Problem Statement
Globalization has led to increasingly diverse speech varieties, yet current spoken dialogue systems frequently fail on under-represented dialects and accents. This misidentification of input language causes cascading failures in downstream tasks, such as Large Language Model (LLM) inference, which rely on accurate transcripts. Existing Automatic Speech Recognition (ASR) systems, including state-of-the-art models like Whisper and Massively Multilingual Speech (MMS), struggle with real-world human speech due to systematic under-representation of regional dialects in training data. Standard fine-tuning approaches are computationally expensive, prone to overfitting on high-dimensional speech data, and often require large datasets that are unavailable for low-resource dialects. Furthermore, existing robustness certification methods typically analyze trained non-convex networks as black boxes, estimating local Lipschitz constants post-hoc rather than providing constructive guarantees.

Methodology: Convex Language Detection (CLD)
The authors propose Convex Language Detection (CLD), a framework that integrates theoretically grounded convex optimization into the ASR pipeline to perform robust language identification before decoding.

1. Convex Reformulation: The core of CLD leverages the convex reformulation of two-layer ReLU networks (based on Pilanci & Ergen, 2020). Instead of optimizing the non-convex objective of standard neural networks, CLD reformulates the problem as a convex program. This is achieved by characterizing all possible ReLU activation patterns induced by the input data. The non-convex training objective is replaced with an equivalent convex program (Eq. 2) that minimizes a loss function plus a regularization term over sampled activation patterns.
2. Optimization via ADMM: To solve this high-dimensional convex program efficiently, the authors implement the Alternating Direction Method of Multipliers (ADMM) in JAX. This allows for multi-GPU parallelization and batched processing, ensuring the solution converges to a global optimum in polynomial time.
3. Integration with ASR: CLD operates as a lightweight detection head. It takes hidden representations from a pre-trained ASR encoder (e.g., Whisper or MMS), processes them through the trained convex module to predict a language token, and feeds this token as an initialization to the decoder. This allows the decoder to generate transcripts conditioned on the correct language, preventing cross-lingual decoding errors.
4. Theoretical Guarantees: The framework provides certified robustness. By analyzing the variation norm of the convex solution, the authors derive exact logit-Lipschitz constants. They prove that the model possesses "certified margin stability," meaning that for any input with a sufficiently large initial classification margin, the predicted label remains invariant within a guaranteed radius of feature perturbations. This bound is data-dependent and computable directly from the learned weights.

Key Contributions

• CLD Framework: A fast, sample-efficient algorithm for robust spoken language classification in low-resource regimes. It demonstrates strong efficiency in the critical task of dialect identification within ASR.
• Convex Recasting and Certification: The authors recast the detection network as a convex program, proving certified robustness against hidden-feature perturbations. They derive exact logit-Lipschitz constants and provide a computable certificate of label invariance, ensuring prediction stability within a guaranteed radius.
• Empirical Validation: Extensive experiments across five languages and twenty-four sub-dialects show that CLD remains performant even with fewer than 100 training samples. In low-resource settings, it achieves 97–98% accuracy, consistently outperforming competitors.
• Open Source: The authors provide a pip-installable JAX package and open-source code to facilitate reproducibility and global inclusivity in speech tools.

Experimental Results
The authors evaluated CLD against baseline ASR models (Whisper-Small, Whisper-Large-v3, MMS-1B) and standard language detection methods (Vanilla Neural Networks, SVMs, KNN).

• Low-Resource Efficiency: In binary classification tasks (English vs. Chinese) with as few as 100 samples per language, CLD achieved a minimum detection accuracy of 96.94% (with 10,000 samples) and up to 99.14% (with 1,000 samples). In contrast, Vanilla Neural Networks (NN) and fine-tuned Whisper models showed a strong correlation between sample size and performance, struggling significantly with small datasets.
• Dialect Robustness: On challenging dialects like Min Dong Chinese and Taiwanese dialects, where baseline models and fine-tuned NNs often failed (e.g., <30% accuracy or misclassifying Chinese as English), CLD achieved accuracies ranging from 88.73% to 100%.
• Multiclass Performance: In a 5-language, 24-dialect multiclass setting, CLD achieved the best performance across all metrics. Compared to the baseline MMS-1B, CLD increased detection accuracy by up to 44.78% and reduced Word Error Rate (WER) by 12.74%.
• Computational Efficiency: CLD demonstrated significant training efficiency, requiring only 64.45 seconds to train on the multiclass dataset (approx. 7.7% of the time required for a Vanilla NN) and 13x fewer TFLOPs. This is attributed to the convex formulation's unique global optimum, which eliminates the need for extensive hyperparameter grid searches.

Significance and Claims
The paper claims that CLD represents the first practical application of convex optimization reformulations specifically for language identification in speech dialogue systems. Its significance lies in:

• Democratizing Access: By functioning effectively with low-resource data, CLD aims to democratize access to spoken dialogue systems for users with diverse multicultural backgrounds and accents, addressing issues of accessibility and inclusion.
• Theoretical Rigor: Unlike heuristic approaches, CLD offers global optimality guarantees and explicit, data-dependent robustness certificates, moving beyond the "black box" nature of standard deep learning robustness analysis.
• Practical Deployment: The method integrates seamlessly into existing ASR pipelines with negligible latency (sub-500ms), making it a viable plug-in module for improving the robustness of large-scale models like Whisper and MMS without retraining the entire encoder-decoder architecture.

The authors conclude that while future work may explore end-to-end differentiable integration of the convex program, the current framework successfully mitigates English-centric failure modes and provides a robust, efficient solution for diverse dialectal speech conditions.