💬 NLP

Best-of- $N$ TTS Evaluation is Confounded by ASR Family Alignment

This paper reveals that Best-of- $N$ TTS evaluation using ASR verifiers is significantly confounded by family-level alignment biases, and proposes cross-family rank ensembles to achieve more robust and accurate content consistency metrics.

Original authors: Taehyung Yu, Seongjae Kang

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Taehyung Yu, Seongjae Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hosting a talent show for a robot voice generator. The robot, let's call it "F5-TTS," tries to read a script perfectly. Sometimes, it stumbles over a word. To fix this, the robot generates 10 different versions of the same sentence (like asking 10 different actors to read the line) and picks the one that sounds the most accurate. This is called Best-of-N.

To decide which version is the winner, you need a judge. In the world of AI voices, this judge is an ASR (Automatic Speech Recognition) system—a robot that listens and writes down what it hears.

The Big Twist: The Judge's Family Tree Matters

The paper's main discovery is a bit like realizing that a sports referee's decision depends entirely on which team they used to play for.

The researchers found that the "quality" of a Best-of-N selection isn't an absolute truth; it changes depending on which family of ASR judge you use. They tested three major families of judges: Whisper, wav2vec 2.0, and HuBERT.

Here's the wild part:

If you use a Whisper judge, it might say, "Version A is the best!"
But if you use a wav2vec judge, it might say, "No way, Version B is the winner!"
And a HuBERT judge might pick yet another one.

The paper shows that if you pick your winner using a Whisper judge, that same winner might actually look worse when a wav2vec judge listens to it. It's as if the judges are biased toward their own "cousins." The paper calls this ASR Family Alignment.

What They Ruled Out (The "It's Not the Voice" Theory)

You might think, "Maybe the judges just hear things differently because their brains (audio encoders) are wired differently?"

The authors tested this by measuring how similar the judges' internal "brains" are using a math tool called Linear CKA. They found that some judges from the same family have nearly identical brains (a similarity score of 0.978, which is almost the same).

However, the paper explicitly rules out the idea that this brain similarity explains the bias.

Even though two Whisper judges have almost identical brains, they don't always agree on the winner.
Conversely, judges with very different brains sometimes agree perfectly.
The pattern suggests the issue isn't about how they hear, but about who they are. It's like a "family loyalty" bias, similar to how a human might trust a friend's opinion more than a stranger's, even if the stranger is smarter. The paper suggests this is a "speech analog of LLM-as-a-judge self-bias."

The Numbers: How Much Does It Matter?

The researchers ran these experiments on a dataset called LibriSpeech-PC test-clean. Here is what the numbers say:

The Baseline: The standard F5-TTS system had a Word Error Rate (WER) of 2.06%. This means it got about 2 words wrong out of every 100.
The "Same-Family" Boost: When they used a Whisper judge to pick the best version, and then checked that version with another Whisper judge, the error rate dropped to 1.72% (a 16.5% improvement).
The "Cross-Family" Problem: But if they used a Whisper judge to pick the winner, and then checked it with a wav2vec judge, the improvement vanished or even got worse.
The Oracle Limit: The researchers calculated the "Oracle" (the absolute best possible choice if you had a magic crystal ball). Even with the best setup, there is still a gap. The Oracle could get the error rate down to 1.42%, meaning there is still room for improvement that current methods aren't reaching.

The Solution: The "Group Hug" Strategy

Since no single judge is perfect, the authors propose a new way to pick the winner: Cross-Family Rank Ensembles.

Instead of asking one judge, they ask judges from different families to rank the 10 versions. Then, they combine the scores.

Rank-Averaging: They take the average rank from a Whisper judge and a wav2vec judge.
Max-Rank: They pick the version that does well enough for both, ensuring no single family dominates.

The Result:
Using this "group hug" method with N=10 candidates, they achieved a mean WER of 1.61% across all three judges. This is a 12% improvement over the baseline. Crucially, this method works well no matter which judge you use to check the final result, whereas picking a single "favorite" judge only works if you check with that same judge's family.

The Takeaway

The paper concludes that if you only report your results using one type of judge (like the official F5-TTS evaluator, which uses Whisper), you might be seeing a "fake" improvement that only exists because the judge and the selector are from the same family.

To be truly sure, the authors recommend Cross-Evaluator Triangulation: always report your results using at least two different ASR families with different training lineages. It's the only way to make sure your robot voice is actually better, and not just better at pleasing a specific type of judge.

Technical Summary: Best-of-N TTS Evaluation is Confounded by ASR Family Alignment

1. Problem Statement

Recent flow-matching zero-shot Text-to-Speech (TTS) systems (e.g., F5-TTS, E2 TTS, CosyVoice 2) achieve high naturalness and speaker similarity but still exhibit non-trivial word-level content errors. The standard inference-time remedy is Best-of-N (BoN) inference, where $N$ candidates are synthesized and the one with the lowest Automatic Speech Recognition (ASR) Word Error Rate (WER) against the target text is selected.

However, the literature lacks systematic attention to the choice of the verifier ASR (used to select the candidate) versus the evaluator ASR (used to report the final WER). This paper identifies a critical confound: verifier rankings are systematically dependent on the ASR family of the evaluator. A verifier that appears superior under one ASR family (e.g., Whisper) may perform poorly under another (e.g., wav2vec 2.0), leading to inconsistent conclusions about model performance based solely on the choice of evaluator.

2. Methodology

The authors conducted a four-way evaluator ablation study using the F5-TTS base model on the LibriSpeech-PC test-clean subset.

Synthesis: Generated 10 BoN candidates per utterance using default F5-TTS settings (32 ODE steps, CFG scale 2.0).
Verifiers: Compared single-model verifiers from three ASR families:
- wav2vec 2.0: w2v2-base (95M params).
- Whisper: distil-sm (166M) and distil-v3 (756M).
- Ensemble: ens3 (rank-average of the three above).
Evaluators: Scored all candidates using four ASR checkpoints spanning three families:
- fwhisper-lgv3 (Whisper-large-v3, official F5-TTS evaluator).
- w2v2-lv60 (wav2vec 2.0 large).
- hubert-lg (HuBERT large).
- distil-sm (used as both verifier and evaluator).
Analysis:
- Measured Oracle Headroom: The gap between single-shot performance and the theoretical best of $N$ candidates.
- Analyzed Representation Similarity: Used linear Centered Kernel Alignment (CKA) on audio encoder representations to test if high similarity predicts ranking agreement.
- Proposed Cross-Family Rank Ensembles: Two aggregation strategies (rank-avg and max-rank) to select candidates by aggregating scores across disjoint ASR families.

3. Key Contributions

Documentation of Evaluator-Dependent Rankings: The authors demonstrate that the same set of generated outputs is ranked in opposing directions by evaluators from different ASR families. A verifier preferred under one family can lose under another, making single-evaluator reports potentially misleading.
Quantification of the Confound: On LibriSpeech-PC, the choice of (verifier, evaluator) family pairing has a larger impact on reported WER than the choice between common verifier checkpoints. Furthermore, same-family verifier-evaluator pairs recover 2–3× more oracle headroom than cross-family pairs, despite having near-identical representations (linear CKA $\approx$ 0.978 for same-family pairs).
Rejection of Representation Similarity: The study rules out audio-encoder representation similarity (linear CKA) as the dominant explanation for the confound. High CKA does not guarantee ranking agreement; instead, the pattern suggests an identity- or lineage-level coupling (analogous to LLM-as-a-judge self-bias), where evaluators disproportionately favor verifiers sharing their training lineage.
Proposed Mitigation Strategies: The paper proposes cross-family rank ensembles (rank-avg and max-rank) that aggregate verifiers across ASR families. It recommends cross-evaluator triangulation (reporting WER under at least two ASR families with disjoint training lineages) as the default reporting practice.

4. Key Results

Rank Reversal: Under the official fwhisper-lgv3 evaluator, the distil-v3 verifier (Whisper family) achieved the lowest WER (1.88%). However, under the w2v2-lv60 evaluator (wav2vec family), the w2v2-base verifier (wav2vec family) ranked highest.
Oracle Recovery: Same-family pairs recovered significantly more potential improvement. For example, under fwhisper-lgv3, distil-v3 recovered 26.0% of the oracle headroom, while the cross-family w2v2-base recovered only 7.9%.
Ensemble Performance:
- Cross-family rank ensembles achieved the lowest mean WER across all three independent evaluators simultaneously.
- At $N=10$ , the rank-avg ensemble achieved a mean WER of 1.61% (a 12% relative reduction vs. F5-TTS baseline).
- The best single verifier (distil-v3 under its own family) reduced WER from 2.06% to 1.72% (−16.5%) but showed degradation or stagnation when evaluated by cross-family ASRs.
Quality Preservation: All BoN configurations maintained speaker similarity (SIM-o) and naturalness (UTMOS) within negligible margins of the baseline, indicating WER gains did not come at a measurable quality cost.

5. Significance and Claims

The paper argues that any BoN comparison reported under a single ASR evaluator is potentially confounded. The primary significance lies in exposing a structural bias in TTS evaluation where "verifier quality" is not an intrinsic property of the model but a function of its alignment with the evaluator's training lineage.

The authors claim that:

Triangulation is Essential: Reporting WER under a single ASR family is insufficient; results must be validated across disjoint ASR families to ensure robustness.
Ensembling is Effective: Cross-family rank ensembles provide the most robust selection strategy, mitigating the "self-bias" of same-family evaluation while maintaining high performance.
Future Work Direction: The substantial gap between current BoN performance and Oracle headroom (e.g., 1.42% vs. 1.61% at $N=10$ ) suggests significant room for improvement in verifier design, potentially through adversarial reweighting against in-family inflation.

The study is limited to the F5-TTS backbone and LibriSpeech-PC, leaving generalization to other TTS backbones and human-MOS triangulation for future work.