How to Evaluate Speech Translation with Source-Aware Neural MT Metrics

1. Problem Statement

Automatic evaluation of Speech-to-Text (ST) translation systems traditionally relies on comparing system outputs against human reference translations using metrics like BLEU. However, this approach ignores the original source audio, inheriting the limitations of reference-based evaluation.

In Machine Translation (MT), source-aware neural metrics (e.g., COMET, MetricX) have demonstrated superior correlation with human judgments by incorporating the source text. Applying these to ST is non-trivial because:

1. Modality Mismatch: The source is audio, not text.
2. Data Scarcity: High-quality, gold-standard transcripts and sentence-level alignments between source audio and reference translations are often unavailable in real-world scenarios.
3. Alignment Issues: Even if synthetic sources are generated, they often lack the precise segmentation alignment required by source-aware metrics.

The paper addresses three core research questions (RQs):

• RQ1: Can synthetic textual sources (derived from audio) replace gold transcripts without compromising metric reliability?
• RQ2: Which synthetic source generation method is better: Automatic Speech Recognition (ASR) transcripts or Back-Translation (BT) of the reference?
• RQ3: How can we handle segmentation mismatches between synthetic sources and reference translations in real-world settings?

2. Methodology

The authors propose a framework to generate synthetic sources and align them with reference translations to enable the use of source-aware metrics (COMET and MetricX) in ST evaluation.

A. Synthetic Source Generation Strategies

Two methods are explored to create a textual proxy for the input audio:

1. ASR Transcripts: Using Automatic Speech Recognition models (Whisper, OWSM, SeamlessM4T) to transcribe the source audio.
   • Pros: Closer to the original spoken content (lexicon/syntax).
   • Cons: Prone to transcription errors (WER); requires re-segmentation; potential bias if the ASR model shares components with the ST system.
2. Back-Translation (BT): Translating the reference target text back into the source language using a Machine Translation model (MADLAD, NLLB).
   • Pros: Computationally cheaper; inherently aligned with the reference segments; neutral regarding the ST system's ASR component.
   • Cons: May introduce artifacts from the reference translation rather than the original speech; depends on MT quality.

B. Cross-Lingual Re-Segmentation (XLR-Segmenter)

Since ASR transcripts are generated from audio segments that rarely match the reference translation segments, a re-alignment step is necessary. The authors introduce a novel two-stage algorithm, XLR-Segmenter:

1. XL-Segmenter (Base): A cross-lingual extension of the standard intra-lingual L-Segmenter. It translates the reference target segments into the source language (via BT) and aligns the ASR transcript to these BT segments using Levenshtein distance minimization.
2. Boundary Refinement (XLR-SimAlign / XLR-LaBSE): To fix misalignments at segment boundaries (where words are unaligned), the algorithm uses semantic embeddings rather than surface string matching.
   • XLR-SimAlign: Uses word embeddings (mBERT) and SimAlign to find optimal split points that minimize cross-alignments.
   • XLR-LaBSE: Uses sentence embeddings (LaBSE) to minimize the cosine distance between split segments.

3. Experimental Setup

• Datasets: Two large-scale ST benchmarks: MuST-C (TED Talks, 14 languages) and Europarl-ST (Parliamentary proceedings, 9 languages). A low-resource pair (Bemba-English) was also tested.
• Systems: Six ST systems covering diverse architectures (Direct vs. Cascaded) and performance levels.
• Metrics: COMET-22 and MetricX-24-Hybrid (selected for high correlation with human judgment).
• Evaluation: Pearson correlation between scores derived from synthetic sources vs. scores derived from gold manual transcripts.

4. Key Results

A. Effectiveness of Synthetic Sources (RQ1)

• Both ASR and BT synthetic sources yield very high correlations (>0.90) with standard metrics using gold transcripts.
• MetricX is more sensitive to source quality than COMET (which showed high correlation even with shuffled sources), making MetricX the primary focus for source-aware analysis.

B. ASR vs. Back-Translation (RQ2)

• ASR Superiority: ASR transcripts generally provide better correlation with gold metrics than BT, provided the ASR Word Error Rate (WER) is ≤ 20%.
• The 20% Threshold:
  • If WER ≤ 20%: ASR is the preferred source (outperforms BT in ~87% of cases).
  • If WER > 20%: BT becomes the superior choice (outperforms ASR in ~86% of cases).
• Bias: Using ASR transcripts generated by a model related to the ST system (e.g., the ASR component of a cascaded system) introduces bias, artificially inflating scores. BT is more neutral in this regard.

C. Re-Segmentation Performance (RQ3)

• The proposed XLR-Segmenter (specifically with the SimAlign refinement) successfully recovers the semantic alignment between synthetic sources and references.
• In "in-the-wild" scenarios (automatic audio segmentation + ASR), the re-segmentation process results in only a 2–3% degradation in semantic similarity (LASER score) compared to manual segmentation, confirming its robustness.

D. Low-Resource and Human Validation

• Low-Resource (Bemba-English): The findings hold true even in low-resource settings where models are weaker.
• Human Judgment: Validation against human Direct Assessment (DA) scores on the Hearing-to-Translate corpus shows that synthetic source-aware metrics correlate better with human judgments than standard reference-only metrics, and even slightly better than standard source-aware metrics using gold transcripts.

5. Key Contributions

1. First Systematic Study: The first comprehensive investigation into using source-aware MT metrics for ST evaluation when gold transcripts are unavailable.
2. XLR-Segmenter Algorithm: A novel, open-source two-stage cross-lingual re-segmentation algorithm that enables robust alignment between ASR transcripts and reference translations.
3. Practical Guidelines: Establishes a clear decision framework for practitioners:
   • Use ASR if the expected WER is < 20% (higher accuracy, closer to original speech).
   • Use BT if ASR quality is poor (WER > 20%) or if computational neutrality/cost is a priority.
4. Empirical Validation: Demonstrates that synthetic source-aware metrics are not just viable but often superior to reference-only metrics in correlating with human quality judgments.

6. Significance

This work bridges a critical gap in Speech Translation evaluation. By proving that synthetic sources (ASR or BT) combined with intelligent re-segmentation can effectively replace gold transcripts, the paper enables the adoption of state-of-the-art, source-aware evaluation metrics in real-world ST scenarios where manual transcripts are too costly or unavailable. This leads to more accurate, principled, and scalable evaluation methodologies for the ST community.