The Unheard Alternative: Contrastive Explanations for Speech-to-Text Models

1. Problem Statement

While Explainable AI (XAI) has made significant progress in text-based Natural Language Processing (NLP), the application of contrastive explanations to Speech-to-Text (S2T) models remains an open challenge.

• Contrastive Explanations: Unlike standard explanations that answer the question "Why did the model produce output P?", contrastive explanations answer the question "Why did the model produce P instead of Q?" (where Q is a contrast object/alternative). This aligns more closely with human reasoning.
• The Gap: Existing explanation methods for S2T (e.g., SPES) generate holistic saliency maps over audio spectrograms. These maps highlight features relevant to the generation of any word but fail to isolate features specific to the choice between two competing outputs (e.g., why the model chose the masculine form curioso instead of the feminine curiosa).
• Specific Challenges in S2T:
  1. Input Modality: Audio signals are multidimensional (time and frequency) and are represented as spectrograms.
  2. Output Granularity: S2T models generate subword tokens, not complete words, making the aggregation of probabilities at the word level non-trivial.
  3. Evaluation: Standard contrast evaluation functions used in text-based NLP often fail when the model strongly prefers the target over the contrast object, resulting in explanations indistinguishable from non-contrastive ones.

2. Methodology

The authors propose the first framework for obtaining contrastive explanations in S2T by adapting the SPES method (Spectrogram Perturbation for Explainable Speech-to-text). The approach involves two critical technical interventions:

A. Aggregation of Word Probability at the Word Level

Since S2T models output subword tokens, the probability of a complete word ($t$) must be reconstructed from its components, the subwords ($w_0, \dots, w_k$).

• Limitation of Previous Work: Simple aggregation methods (averaging, max-pooling, or naive chain rule multiplication) neither account for mismatches in token count between the target and the contrast object nor the fact that a subword sequence might be a prefix of a longer word rather than a complete word.
• Proposed Solution: The authors adopt the Word Boundary Method (Pimentel and Meister, 2024). This method calculates the probability of a word sequence by ensuring it is followed by a "beginning of word" token or a punctuation token. This prevents the overestimation of probabilities for subword sequences that are merely prefixes (e.g., distinguishing between _professor and _professor e ssa).

B. Contrast Evaluation Functions

The core of the method is an evaluation function that quantifies how perturbing an input region affects the probability of the target ($t$) compared to the contrast object ($f$).

• The Problem with the Difference Scorer ($S_{CD}$): The standard contrast scorer, $S_{CD}(t, f) = (p(t) - \tilde{p}(t)) - (p(f) - \tilde{p}(f))$, often collapses to the baseline scorer $S_B(t)$ when $p(t) \gg p(f)$. In such cases, the explanation highlights features relevant to generating any correct word, not the specific contrast.
• Proposed Solution: The authors introduce the Relative Scorer ($S_{CR}$), based on Jacovi et al. (2021):
  $$S_{CR}(t, f) = \frac{p(t)}{p(t) + p(f)} - \frac{\tilde{p}(t)}{\tilde{p}(t) + \tilde{p}(f)}$$
  This function normalizes the contribution of both terms by their sum. It ensures that even when the target is much more probable than the contrast object, the scorer remains sensitive to changes in the contrast object's probability, thereby effectively isolating features specific to the contrast.

C. Evaluation Metrics

To validate the faithfulness of the explanations (how well they reflect actual model behavior), the authors adapt metrics from text classification:

1. Coverage: The percentage of cases where the model generates either the target or the contrast object after deleting features. High coverage ensures that the identified features are specific to the contrast and not to general text generation.
2. Flip Rate: The rate at which the model switches from predicting the target to the contrast object when salient features are progressively masked (deleted). A faithful explanation should induce a high flip rate with minimal deletion.

3. Main Contributions

1. First S2T Contrastive Method: Introduction of a novel pipeline for generating contrastive explanations for speech models by adapting perturbation-based attribution to spectrograms.
2. Novel Evaluation Mechanism: Proposal of the Relative Scorer ($S_{CR}$) as a superior alternative to the standard difference scorer for S2T, demonstrating that it produces distinct, truly contrastive saliency maps.
3. Robust Evaluation Framework: Development of a dual-metric evaluation (Coverage + Flip Rate) tailored to generative S2T tasks to measure explanation faithfulness.
4. Empirical Validation: Validation of the method in assigning speaker gender in speech translation across three language pairs (English $\to$ French, Italian, Spanish).

4. Results

The method was evaluated using the MuST-SHE dataset, focusing on translating gender-neutral English terms into gender-specific Romance languages (e.g., "I am curious" $\to$ curioso vs. curiosa).

• Differentiation of Saliency Maps:
  • Difference Scorer ($S_{CD}$): Generated saliency maps that were highly correlated (Pearson $r \approx 0.93$) with non-contrastive explanations. It failed to isolate gender-specific features.
  • Relative Scorer ($S_{CR}$): Generated maps with low correlation ($r \approx 0.33$) to non-contrastive maps and successfully isolated acoustic regions relevant to gender.
• Coverage Performance:
  • When deleting features identified by the Difference Scorer, coverage dropped below 20% after only 2% deletion (indicating the features were generic).
  • The Relative Scorer maintained coverage of >30% even after 20% deletion, proving it targets features specific to the gender contrast.
• Flip Rate Performance:
  • Feminine Targets: Masking the top 5% of features identified by the Relative Scorer caused the model to switch to masculine predictions in >70% of covered cases.
  • Masculine Targets: Flip rates were lower (~30%), which the authors attribute to a masculine default bias in S2T models (where masculine is the default unless strong feminine cues are present), rather than a failure of the method.
• Model Robustness: Results remained consistent across different architectures (Multilingual Transformer, Monolingual Conformer) and language pairs.

5. Significance and Implications

• Advancing XAI for Speech: This work bridges the gap between text-based techniques for contrastive explanations and the unique challenges of speech processing, providing a foundation for understanding how S2T models utilize acoustic cues.
• Bias Detection: The method effectively isolates the acoustic features (e.g., pitch, formants) that models use for gender disambiguation. This is crucial for examining gender bias in AI, as it reveals that models often rely on vocal features that can disadvantage transgender speakers or individuals with atypical vocal characteristics.
• Future Applications: Although demonstrated on gender, the methodology is transferable to other S2T phenomena, such as homophone disambiguation, coreference resolution, and politeness register selection.
• Ethical Awareness: The paper explicitly acknowledges the ethical risks of using vocal features for gender prediction, noting that while this improves accuracy, it can harm marginalized groups. The tool was designed to explain current model behavior to enable better oversight and future mitigation strategies.

In summary, the paper offers a rigorous technical framework to understand why S2T models make specific decisions between alternatives, moving beyond what they predict to how they decide, with a specific focus on the critical issue of gender bias in speech translation.