Technical Summary: Towards Out-of-Distribution Detection in Vocoder Recognition via Latent Feature Reconstruction

Problem Statement

The rapid advancement of speech synthesis and voice conversion technologies has enabled the generation of highly natural, human-like speech, creating significant security risks regarding audio impersonation and deepfakes. A critical challenge in detecting these synthetic audio samples is Out-of-Distribution (OOD) detection. Current deepfake recognition systems often struggle because classification models cannot cover all existing or future vocoder algorithms.

Existing approaches to OOD detection in this domain primarily rely on:

1. Probability-score-based methods: Assuming in-distribution (ID) samples have higher maximum softmax probabilities than OOD samples.
2. Classify-distance-based methods: Classifying samples far from the center of ID classes as OOD.

The authors argue that these methods have inherent limitations. Probability-based approaches struggle with outliers because models are trained exclusively on ID data. Distance-based methods can suffer from accuracy degradation at threshold edges. Furthermore, these two-class methods often require careful threshold selection and fail to explicitly consider the characteristic tendencies of the audio generation process itself.

Methodology

The paper proposes a reconstruction-based detection approach utilizing an autoencoder architecture to address OOD detection in vocoder recognition. The system operates on the premise that acoustic features specific to a known vocoder class can only be effectively reconstructed by a decoder trained on that specific class.

System Architecture

1. Feature Extraction: The system uses a pre-trained WavLM model (frozen during training) to extract acoustic features from input audio. The authors investigate various intermediate layers of WavLM, finding that the 6th layer or a weighted combination of layers up to the 18th yields optimal results.
2. Encoder-Decoder Structure:
   • Encoder: Compresses the input acoustic feature into a lower-dimensional hidden representation. It consists of three transformer modules (convolutional layer, transformer encoder block, linear layer).
   • Multiple Decoders: There is one decoder for each known ID vocoder class. Each decoder attempts to reconstruct the hidden feature back to the original input space.
3. Inference Logic:
   • During inference, the input feature is passed through all decoders.
   • The Mean Square Error (MSE) between the original input and the reconstructed output is calculated for each decoder.
   • ID Classification: If the MSE for one specific decoder is below a predefined threshold (while others are high), the sample is classified as ID corresponding to that decoder's class.
   • OOD Classification: If the MSE between the input and all decoders exceeds the threshold, the sample is classified as OOD.

Enhancements and Constraints

To address the indistinguishability between ID and OOD samples and ensure decoders learn class-specific representations, the authors introduce two auxiliary constraints:

1. Contrastive Learning: Applied to the reconstructed features to minimize the distance between the input and its correct reconstruction (positive pair) while maximizing the distance between the input and reconstructions from other decoders (negative pairs).
2. Auxiliary Classifier: A classifier attached to the encoder's output to ensure the compressed latent representation aligns closely with the specific vocoder class before reconstruction. This prevents the encoder from generating similar outputs for different classes.

The total training loss is a weighted sum of the reconstruction loss (MSE), contrastive loss, and classification loss.

Key Contributions

• Reconstruction-Based Paradigm: The paper shifts from probability/distance-based OOD detection to a reconstruction-based approach, explicitly modeling the characteristic tendencies of the speech synthesis process.
• Class-Specific Decoding: The architecture employs multiple decoders, where each is specialized for a specific vocoder class, ensuring that features are only well-reconstructed by their corresponding decoder.
• Dual Constraint Mechanism: The integration of contrastive learning and an auxiliary classifier significantly enhances the distinctiveness of reconstructed features, solving the issue of similar outputs from different decoders.
• Feature Layer Investigation: The study systematically evaluates different intermediate layers of the WavLM model, identifying that combining outputs from layers up to the 18th (weighted) provides superior performance compared to single-layer outputs.

Experimental Results

The proposed method was evaluated on the WaveFake dataset, which contains audio generated by seven known vocoder architectures (ID samples). OOD samples were generated using unseen models (BigvGAN and UnivNet).

• Performance: The proposed system achieved an F1 score of 68.04% on the evaluation dataset, outperforming baseline systems (ECAPA-TDNN, AASIST, RawNet2) which scored between 63.15% and 64.08%. This represents a relative improvement of approximately 10% over the baselines.
• Ablation Studies:
  • Removing the contrastive loss caused a significant drop in all metrics (F1 dropped to 59.21%), confirming its role in separating class instances.
  • Removing the auxiliary classifier also reduced performance (F1 dropped to 55.48%), validating its necessity in constraining the encoder's latent space.
• Visualization: MSE analysis showed a distinct boundary between ID and OOD samples, where OOD samples consistently exhibited higher reconstruction errors across all decoders.

Significance

The paper claims that its reconstruction-based approach offers a more robust solution for deepfake algorithm recognition by directly addressing the limitations of probability and distance thresholds. By focusing on the reconstruction capability of class-specific decoders and enforcing distinctiveness through contrastive learning and auxiliary classification, the method effectively identifies samples that do not belong to any known vocoder distribution. The results suggest that this approach provides a superior safety protection solution for detecting synthetic speech, particularly in scenarios involving unknown or unseen generation algorithms.