Technical Summary: Cross-Modal Robustness Transfer (CMRT)

Problem Statement

End-to-End Speech Translation (E2E-ST) models have achieved significant performance, often surpassing cascaded Automatic Speech Recognition (ASR) and Machine Translation (MT) pipelines. However, current models are primarily evaluated on curated, "clean" datasets (e.g., MuST-C, CoVoST 2), which fail to capture the complexity of real-world non-native and dialectal speech. Specifically, these models lack robustness to inflectional morphological variations (e.g., incorrect verb conjugations or noun declensions) common in spontaneous non-native speech.

While adversarial training using synthetic text data has improved robustness in Neural Machine Translation (NMT), applying similar techniques to E2E-ST is hindered by the prohibitive computational cost and technical difficulty of generating high-quality adversarial speech data. Synthesizing adversarial audio via Text-to-Speech (TTS) introduces acoustic artifacts and requires high-fidelity TTS models for every source language, creating a significant barrier to training robust speech systems.

Methodology

The authors propose Cross-Modal Robustness Transfer (CMRT), a framework designed to transfer adversarial robustness from the text modality to the speech modality, eliminating the need for adversarial speech data during training. The method operates on the hypothesis that if speech and text representations are sufficiently aligned in a shared semantic space, the model's sensitivity to perturbations in one modality can be mitigated using the other.

The approach consists of two primary stages:

1. Semantic Alignment (CMRT-TR)

The first stage focuses on bridging the modality gap to create a shared latent space where speech and text embeddings are highly correlated. This involves:

• Architecture: A standard E2E-ST model comprising a speech encoder (HuBERT or mHuBERT), a text embedding layer, and a translation encoder-decoder.
• Word-Aligned Contrastive Learning (WACO): Using forced alignment between speech and text, the model maximizes the cosine similarity between corresponding word-level speech and text embeddings.
• Mixup Training: The model is trained on a mixture of speech and text embeddings for individual words. This familiarizes the translation encoder-decoder with receiving mixed-modality inputs, a prerequisite for the subsequent robustness transfer.
• Loss Functions: The training objective combines Speech Translation (ST) loss, Machine Translation (MT) loss, Contrastive Loss (WACO), and Cross-Entropy on mixup inputs, regularized by symmetric Kullback-Leibler (KL) divergence to align output distributions.

2. Robustness Fine-Tuning (CMRT-FN)

In the second stage, the model is fine-tuned to handle adversarial perturbations without generating new audio.

• Speech-MORPHEUS: The authors adapt the MORPHEUS framework (originally for text) to generate inflectional adversarial examples. This tool replaces words with morphologically incorrect but semantically plausible alternatives (e.g., changing "happiness" to "happy" in a context requiring a noun).
• Adversarial Mixup: Instead of synthesizing adversarial speech, the method injects adversarial text embeddings directly into the speech manifold. The speech encoder is frozen to preserve the alignment learned in Stage 1.
• Training Strategy: During fine-tuning, the model receives clean speech inputs but is forced to process adversarial text embeddings for specific words identified by the attack. The model learns to map these adversarial text representations to the correct translation.
• Asymmetric KL Divergence: The loss function uses asymmetric KL divergence to minimize the difference between the output distribution of the adversarial mixup and the clean input distributions (both speech and text), effectively training the model to be invariant to the noise.

Key Contributions

1. Speech-MORPHEUS: An adaptation of the MORPHEUS adversarial technique to the speech domain. The authors demonstrate that state-of-the-art E2E-ST models are highly vulnerable to inflectional variations, with performance dropping significantly when tested on these adversarial examples.
2. Cross-Modal Robustness Transfer (CMRT): A novel fine-tuning paradigm that transfers robustness from text to speech. By injecting adversarial text perturbations into a shared latent space, the method significantly improves E2E-ST performance on non-standard inputs without requiring adversarial speech data.
3. Modality Alignment for Robustness: The work establishes that combining contrastive learning (WACO) and mixup training creates the strongest semantic alignment between speech and text, which is a critical prerequisite for successful cross-modal robustness transfer.

Experimental Results

The authors evaluated CMRT on four language pairs: English-German (En-De), English-Catalan (En-Ca), English-Arabic (En-Ar), and French-English (Fr-En) using the CoVoST 2 dataset.

• Robustness Improvement: CMRT-FN improved adversarial robustness by an average of 3.4 BLEU points compared to the baseline HuBERT-Transformer, and 3.1 BLEU points compared to the aligned CMRT-TR model.
• Comparison to Synthetic Speech: While fine-tuning on 50k examples of synthetic adversarial speech (TTS-Morpheus-FN) yielded slightly higher robustness (approx. 1.4 BLEU points better than CMRT-FN), it resulted in a significant performance drop (approx. 3.6 BLEU points) on the original clean test set. In contrast, CMRT-FN maintained competitive performance on the clean test set, with only a marginal drop (0.6 BLEU points) from its aligned baseline.
• Efficiency: The CMRT approach avoids the computational overhead of generating large-scale adversarial audio. The fine-tuning stage (CMRT-FN) required only 1/8th of the time of the initial training stage, whereas generating equivalent TTS data was significantly more time-consuming.
• Representation Analysis: Centered Kernel Alignment (CKA) analysis revealed that the internal representations of the CMRT-FN model converge toward those of models trained on actual adversarial speech, confirming that robustness is effectively transferred through the shared latent space.

Significance and Claims

The paper claims that CMRT establishes a new baseline for robust E2E-ST by overcoming the "robustness-accuracy" trade-off often seen when training on synthetic data. The authors assert that their method effectively addresses the vulnerability of speech translation models to morphological errors without the prohibitive costs associated with generating adversarial audio.

The significance lies in demonstrating that adversarial robustness can be transferred across modalities through shared latent spaces. By leveraging the relative ease of generating adversarial text and the power of semantic alignment, the method provides a scalable solution for improving the real-world reliability of speech translation systems, particularly for non-native speakers.

Limitations

The authors acknowledge two main limitations:

1. Training Complexity: The method requires a two-stage training process (alignment followed by fine-tuning), which increases the complexity of the training pipeline compared to single-stage approaches.
2. Alignment Trade-off: In some language directions (En-X), the initial alignment stage (CMRT-TR) resulted in slightly lower performance on the clean CoVoST 2 test set compared to the standard HuBERT-Transformer baseline. The authors plan to investigate this negative effect in future work.