Technical Summary: Traceback Translators Against Forgetting in Continual Fake Speech Detection

Problem Statement

The rapid advancement of generative models for synthetic speech has necessitated the continuous updating of deepfake detectors. However, standard continual learning strategies face a critical challenge: catastrophic forgetting. When models are fine-tuned on new datasets (e.g., new deepfake generators or languages) without access to original training data, they often lose the ability to detect previously seen attack types. Existing solutions, such as full retraining or simple domain adaptation (e.g., fine-tuning Batch Normalization layers), struggle to balance high performance on new data with the preservation of accuracy on legacy data. Furthermore, many approaches require retraining large portions of the model, leading to high computational costs.

Methodology

The authors propose a forgetting-resilient continual learning framework that utilizes a Traceback Translator within a frozen detector architecture. The core methodology involves the following components:

1. Backbone and Pre-processing

• Architecture: A customized ResNet18 is employed as the backbone detector. The first convolutional layer is frozen, and the classifier head consists of two fully connected layers with Batch Normalization (BN) and Dropout.
• Input: The model processes Mel-frequency cepstral coefficient (MFCC) spectrograms (128 frequency coefficients over 42 time frames), focusing on salient instants where the signal transitions from silence to voiced intervals.
• Loss Function: The classifier uses a weighted Cross-Entropy loss with label smoothing to reduce over-confidence. An additional penalty term is introduced to specifically mitigate the misclassification of real samples as synthetic (specifically class 6), which was observed as a tendency in preliminary experiments.

2. Continual Learning Strategies

The paper evaluates three distinct approaches for adapting to new datasets:

• Full Retraining: Retraining the entire network on new data. While effective for the new task, this leads to severe performance degradation on the source domain.
• Domain Adaptation: Selectively retraining only the Batch Normalization (BN) layers while keeping the rest of the model frozen. This reduces computational load but still results in significant forgetting of the source domain knowledge.
• Domain Translation (Proposed): This approach introduces a lightweight translator network inserted after the embedding layer (128-dimensional latent space) and before the classifier.
  • Mechanism: The translator remaps feature distributions from the new (target) domain to align with the original (source) domain.
  • Training Objective: The translator is trained using a composite loss function:
    1. Classifier Loss: Standard classification loss.
    2. CORAL Loss: Minimizes the difference in mean and covariance between source and target feature distributions.
    3. Prototype Consistency (PC) Loss: Minimizes the intra-class distance between source and target class prototypes (means of real and fake features).
  • Constraint: The backbone ResNet18 remains frozen during this process, ensuring that previously learned representations are not altered.

3. Data Augmentation

To address class imbalance and limited sample sizes for certain deepfake classes, the authors employ a diffusion-based generative process (using a 2D U-Net) to generate synthetic spectrogram samples, alongside classic data augmentation techniques.

Key Contributions

1. Novel Continual Learning Approach: The introduction of a lightweight, domain translator within a frozen classifier to adapt to new deepfake generators and sampling setups without retraining the backbone.
2. Efficiency and Accuracy Trade-off: A comparative analysis demonstrating that the translator-based approach minimizes the number of retrained parameters while maximizing accuracy on both new and old datasets, outperforming traditional fine-tuning and partial retraining methods.
3. Cross-Lingual Validation: Validation of the approach in a multilingual setup (English and Chinese), demonstrating its capability for cross-lingual adaptation.
4. Generative Augmentation: Integration of diffusion models to improve model generalization and performance in the presence of class imbalance.

Experimental Results

The study utilized several benchmark datasets: ASVspoof 2019 (English, source), FakeOrReal (FoR), In-The-Wild (ITW), and ADD 2022 (Chinese).

• Backbone Selection: The customized ResNet18 was selected as the optimal backbone, offering the best balance of accuracy (0.994 Dev. accuracy) and computational efficiency compared to larger models like AST or ConvNeXT.
• Performance Comparison:
  • Full Retraining: Achieved excellent performance on new datasets (e.g., 0.28% EER on ITW) but caused catastrophic forgetting on the source dataset (ASV19), dropping AUC by 52% and increasing EER by 52.9% on average.
  • Domain Adaptation (BN layers): Improved robustness over full retraining but still suffered significant accuracy loss on the source domain (38% AUC decrease).
  • Domain Translation: Achieved a superior trade-off. It maintained the source domain accuracy (95.4% AUC, 9.74% EER) while achieving strong performance on new domains (e.g., 98.1% AUC on ADD22).
• Parameter Efficiency: The proposed method required training only 21K parameters, a negligible fraction compared to the 11M parameters of the full model or the 500K+ parameters required by other continual learning baselines (e.g., CL ALL in [11]).
• Ablation Studies:
  • Diffusion-based data augmentation improved AUC from 91.5% to 95.0%.
  • The combination of CORAL and Prototype Consistency losses proved more effective than using either loss individually.

Significance and Claims

The paper claims that the proposed Traceback Translator strategy effectively mitigates catastrophic forgetting in audio deepfake detection. By decoupling the adaptation process from the feature extraction backbone, the method allows detectors to evolve with new generative threats without sacrificing the ability to detect historical attacks. The authors highlight that this approach is particularly significant for scenarios where access to original training data is unavailable and computational resources are constrained. The results suggest that this lightweight, frozen-backbone architecture is a viable and efficient alternative to full retraining or complex continual learning frameworks, offering a robust solution for the dynamic landscape of synthetic speech detection.