QAMO: Quality-aware Multi-centroid One-class Learning For Speech Deepfake Detection

Technical Summary: QAMO – Quality-aware Multi-centroid One-class Learning for Speech Deepfake Detection

Problem Statement
Traditional speech deepfake detection (SDD) often treats the task as a binary classification problem, which can lead to overfitting on known spoofing attacks and poor generalization to unseen deepfakes. One-class learning (OCL) has emerged as an alternative, modeling a compact distribution of bona fide (real) speech around a single centroid and treating deviations as potential fakes. However, conventional OCL oversimplifies the bona fide class by assuming a single unimodal distribution. This fails to capture the inherent intra-class variability of genuine speech, particularly regarding speech quality (e.g., clarity, presence of noise, realism of prosody). Existing research indicates that significant gaps in Mean Opinion Score (MOS) exist between real and synthetic speech, and that quality variations can influence the separation between bona fide and spoofed samples. Current OCL methods do not explicitly account for these quality subspaces.

Methodology: QAMO Framework
The authors propose QAMO, a framework that extends OCL by introducing multiple centroids, each optimized to represent distinct speech quality subspaces.

1. Quality-Aware Multi-Centroid Modeling:

   • Instead of a single centroid, QAMO learns $Q$ centroids ($w_q$), where each corresponds to a discrete quality level (e.g., low vs. high MOS).
   • Quality levels are assigned to training samples using a pre-trained MOS predictor (Scoreq). The continuous MOS range is thresholded into discrete classes (e.g., $q \in \{0, 1\}$).
   • To ensure centroids are discriminative and aligned with specific quality levels, the model employs an AM-Softmax loss ($L_{quality}$) for quality classification. This forces embeddings of the same quality level to cluster around their specific centroid.

2. One-Class Learning with Multiple Centroids:

   • The framework extends the OC-Softmax loss to a multi-centroid setting.
   • For bona fide inputs, the model computes the cosine similarity distance to the centroid corresponding to the sample's quality level.
   • For spoof inputs, the model penalizes the maximum cosine similarity across all centroids to ensure separation from the entire bona fide subspace.
   • The final training objective combines the QAMO loss ($L_{QAMO}$) and the quality classification loss ($L_{quality}$) with a weighting hyperparameter $\lambda$.

3. Inference Strategy:

   • Max-Score Inference: Similar to prior multi-centroid works, this takes the maximum similarity across all centroids. However, the authors note this acts as a hard assignment, which can be sensitive to quality estimation errors.
   • Ensemble-Score Inference (Proposed): To improve stability and handle uncertainty, QAMO computes a weighted sum of similarities to all centroids. The weights ($\alpha_q$) are derived via a softmax function applied to the cosine similarities. This strategy aggregates evidence across all quality levels, producing smoother, lower-variance scores without requiring explicit quality labels at inference time.

Key Contributions

• Multi-Centroid Quality Modeling: Unlike previous multi-centroid approaches (e.g., SAMO) that organize centroids by speaker identity, QAMO structures them by speech quality levels, explicitly encoding quality information into the bona fide representation.
• Quality-Aware Loss Function: The integration of a quality classification objective ensures that the learned centroids remain distinct and do not collapse into a single point, effectively preserving intra-class variability.
• Robust Inference Mechanism: The proposal of a softmax-weighted ensemble scoring strategy allows the system to operate without quality labels during inference while mitigating the risks of hard assignment and overconfident predictions.
• Data Augmentation Strategy: The authors treat augmented data (via RawBoost) as "low quality" during training, a compute-efficient proxy consistent with observed MOS shifts, rather than ignoring quality degradation.

Experimental Results
The authors evaluated QAMO on the ASVspoof2019 LA training set and tested on ASVspoof2021 LA, ASVspoof2021 DF, In-the-Wild (ITW), and Fake-or-Real (FoR) datasets.

• Performance: When integrated with the XLSR-Conformer-TCM backbone, QAMO achieved an Equal Error Rate (EER) of 5.21% on the In-the-Wild dataset, outperforming the single-centroid OC-Softmax baseline (6.72%) and prior quality-aware systems.
• Comparisons: On the 21DF dataset, QAMO achieved a 1.63% EER, and on FoR, it achieved 3.45%, consistently outperforming baselines including OC-Softmax and weighted cross-entropy (WCE) models.
• Ablation Studies:
  • Removing the quality classification loss ($L_{quality}$) caused performance degradation and centroid collapse, confirming the necessity of explicit quality supervision.
  • The ensemble-score inference strategy significantly outperformed the max-score strategy, particularly on the ITW dataset, by providing better-calibrated scores and clearer separation between bona fide and spoof distributions.

Significance and Claims
The paper claims that QAMO addresses the oversimplification of the bona fide class in traditional one-class learning by explicitly modeling speech quality subspaces. By doing so, it preserves intra-class variability while maintaining discriminability against spoofed audio. The authors assert that this approach leads to more robust detection of unseen deepfake attacks across diverse acoustic conditions. The ensemble scoring strategy is highlighted as a key factor in stabilizing decision boundaries and improving detection robustness without the computational overhead of estimating quality labels during inference. The work underscores the importance of incorporating speech quality metrics to build more reliable countermeasures against evolving deepfake threats.