⚡ electrical engineering

Rehearsal with Auxiliary-Informed Sampling for Audio Deepfake Detection

This paper proposes Rehearsal with Auxiliary-Informed Sampling (RAIS), a continual learning approach for audio deepfake detection that utilizes a label generation network to guide diverse sample selection, thereby mitigating bias and forgetting while achieving superior performance across evolving attack scenarios.

Original authors: Falih Gozi Febrinanto, Kristen Moore, Chandra Thapa, Jiangang Ma, Vidya Saikrishna, Feng Xia

Published 2026-01-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Falih Gozi Febrinanto, Kristen Moore, Chandra Thapa, Jiangang Ma, Vidya Saikrishna, Feng Xia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a club trying to spot fake IDs. At first, you only see fakes from one specific country (Experience 1). You get really good at spotting those. But then, the criminals start using fakes from a different country, then another, and another (Experiences 2, 3, 4, and 5).

If you just keep studying the new fakes, you might forget how to spot the old ones. This is called "catastrophic forgetting." If you try to relearn everything from scratch every time a new fake appears, it takes too long and is too expensive.

This paper introduces a smart system called RAIS (Rehearsal with Auxiliary-Informed Sampling) to help the security guard (the AI model) learn new tricks without forgetting the old ones.

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Note" Memory

Most current systems try to remember the past by keeping a small "memory buffer" (a tiny notebook) of old examples. They pick which old examples to keep using a simple rule: "Keep a mix of Real and Fake."

The paper argues this is like trying to remember a whole library of books by only looking at the cover color (Red for Fake, Blue for Real). You might end up with a notebook full of "Red" books that all look exactly the same, missing the subtle differences between them. When a new type of fake ID shows up that looks a bit like those specific "Red" books, your system gets confused because it never learned the variety within the "Fake" category.

2. The Solution: The "Secret Translator" (AAGM)

To fix this, the authors created a special helper module called the Audio Auxiliary Label Generation Module (AAGM).

Think of the main AI as a bouncer who only cares about the big question: "Is this person real or fake?"
The AAGM is like a secret translator standing next to the bouncer. It doesn't just look at "Real vs. Fake." It listens to the voice and invents its own secret categories, like "Voice sounds like a robot," "Voice sounds like a whisper," or "Voice sounds like a singer."

How it learns: It doesn't need a human to teach it these categories. It figures them out on its own by playing a "fill-in-the-blank" game with the audio (masking parts of the sound and guessing what they are).
The Safety Rule: Crucially, this translator works in a way that doesn't distract the bouncer. It learns its own secret categories without messing up the bouncer's main job of spotting fakes.

3. The Strategy: The "Smart Selection" (AIS)

Now that the system has these secret categories, it uses a new strategy called Auxiliary-Informed Sampling (AIS) to fill its memory notebook.

Instead of just grabbing random "Fake" examples, the system looks at the secret categories. It asks:

"Do I have a 'robot voice' fake in my notebook?"
"Do I have a 'whispering fake'?"
"Do I have a 'singing fake'?"

If the notebook is missing a "robot voice" fake, the system prioritizes saving one. This ensures the memory buffer is diverse. It's like making sure your emergency kit has a flashlight, a first-aid bandage, and a rope, rather than just 10 flashlights.

4. The Results: The "All-Star Team"

The authors tested this system against other methods using five different rounds of new fake audio attacks (from different languages and sources).

The Old Way: Other methods either forgot the old fakes or got confused by the new ones.
The RAIS Way: By keeping a diverse, well-rounded memory of the past, RAIS stayed sharp. It achieved an average error rate of 1.953%, which is incredibly low and nearly as good as if the system had been trained on all the data at once (which is usually impossible in real life).

Summary

In short, this paper says: Don't just remember the past; remember the variety of the past.

By letting the AI invent its own detailed labels for audio characteristics (like a secret translator) and using those labels to pick the most interesting examples to remember, the system becomes much better at spotting new, evolving deepfakes without forgetting what it already learned.

Technical Summary: Rehearsal with Auxiliary-Informed Sampling (RAIS) for Audio Deepfake Detection

1. Problem Statement

Existing audio deepfake detection frameworks suffer from performance degradation when encountering new deepfake attacks. While Continual Learning (CL) offers a solution by allowing models to integrate new information without forgetting prior knowledge, current approaches face significant limitations:

Regularization-based methods (e.g., EWC, LwF) often assume no access to past data, which can introduce bias toward newly observed data.
Rehearsal-based methods (e.g., Experience Replay) rely on storing past samples in a memory buffer. However, existing sample selection strategies often depend solely on primary labels (fake vs. bona fide).
The Core Challenge: Audio data contains diverse paralinguistic features. Ignoring these features during sample selection leads to a lack of diversity in the memory buffer, introducing bias and increasing the risk of catastrophic forgetting.

2. Methodology: RAIS Framework

The authors propose Rehearsal with Auxiliary-Informed Sampling (RAIS), a rehearsal-based CL approach designed to enhance sample diversity and retention. The framework consists of two jointly trained modules and a novel sampling strategy.

2.1. Core Modules

Audio Deepfake Detection Module (ADDM):
- A standard classification network comprising a feature extractor ( $g$ ) and a classification head ( $c$ ).
- It processes input audio $x$ to produce logits and a probability distribution for the primary task: distinguishing between fake ( $y=0$ ) and bona fide ( $y=1$ ) audio.
- Optimized using cross-entropy loss ( $L_{ADDM}$ ).
Audio Auxiliary Label Generation Module (AAGM):
- Designed to infer latent, diverse audio characteristics without manual labeling.
- Architecture: It takes a detached latent representation $z$ from the ADDM's feature extractor and passes it through a label-generation head ( $h$ ) to produce a vector of $K$ possible auxiliary labels.
- Training Objective: The AAGM is trained independently using a masked prediction objective to prevent conflicting gradients with the primary task (achieved via a stop-gradient mechanism).
  - Masked Branch: Enforces distinct auxiliary labels for fake and bona fide samples. A binary mask ensures fake samples only predict from the first $K/2$ labels and bona fide samples from the remaining $K/2$ .
  - Loss Function: Combines Mean Squared Error (MSE) to align masked and unmasked probability vectors, and a Diversity Loss (KL Divergence) to ensure a uniform distribution of generated labels, preventing trivial solutions.
- The final auxiliary label is determined as $\hat{y}_{aux} = \arg\max(p_{mask})$ .

2.2. Auxiliary-Informed Sampling (AIS)

RAIS introduces a new strategy for selecting samples to populate the fixed-size memory buffer $M$ :

Stratification: The dataset is partitioned into primary categories (fake/bona fide) and further subdivided into groups based on the generated auxiliary labels.
Importance Scoring: Samples are scored based on a combination of the primary classification confidence ( $p_{\hat{y}}$ ) and the auxiliary label confidence ( $p_{mask}^{\hat{y}_{aux}}$ ):
$s = \frac{1}{2}(p_{\hat{y}} + p_{mask}^{\hat{y}_{aux}})$
Selection Process:
- The buffer is updated after each experience $E_i$ by allocating space $L = |M|/(i+1)$ .
- AIS performs stratified selection using a round-robin approach across auxiliary label groups within each primary category.
- This ensures that the memory buffer retains a balanced representation of diverse audio characteristics, not just the most common or easily classified samples.

3. Key Contributions

The paper outlines four primary contributions:

RAIS Framework: A rehearsal-based CL approach specifically for audio deepfake detection that improves sample diversity in the memory buffer.
Auxiliary Label Generation: An automated module (AAGM) that infers diverse auxiliary labels via masked prediction, eliminating the need for manual annotation of paralinguistic features.
AIS Strategy: A novel sampling mechanism that leverages these generated labels to select diverse and informative samples, ensuring balanced representation.
Empirical Superiority: Extensive experiments demonstrating that RAIS outperforms state-of-the-art CL methods, achieving the lowest average Equal Error Rate (EER).

4. Experimental Results

The authors evaluated RAIS across five sequential experiences ( $E_0$ to $E_4$ ) using diverse datasets: ASVspoof 2019 LA, VCC 2020, InTheWild, CFAD, and a custom dataset generated via OpenAI TTS.

Performance: With a memory buffer size of 512, RAIS achieved an average EER of 1.953%, closely matching the "Train-on-all" upper bound (1.772%) and significantly outperforming other CL methods.
Comparison: RAIS surpassed regularization-based methods (EWC, LwF, RAWM, RWM) and other rehearsal strategies (ER, ER-ACE with various sampling techniques like Herding, Reservoir, and Class-balanced).
Ablation Studies: Removing key components degraded performance:
- Removing AAGM (-AAGM) increased average EER.
- Removing AIS (-AIS) increased average EER.
- Removing Diversity Loss (-DL) increased average EER.
Parameter Sensitivity: The auxiliary label size $K$ was tested from 10 to 100. $K=90$ yielded the best performance.
Forgetting Analysis: In a controlled experiment focusing on E1 and E2, RAIS demonstrated the lowest forgetting rate (0.17%), outperforming the second-best baseline (ER with Herding at 0.33%).

5. Significance and Claims

The paper claims that RAIS addresses the critical limitation of existing rehearsal methods: the inability to capture the full diversity of audio characteristics. By automatically generating and utilizing auxiliary labels, RAIS ensures that the memory buffer contains a representative subset of past experiences, effectively mitigating catastrophic forgetting.

The authors position RAIS as a highly efficient solution, requiring only a 512-sample buffer (~50 MB) compared to storing full datasets (>50 GB), while achieving near-optimal performance. They note that while the method prioritizes fake samples to reduce privacy risks, the storage of genuine audio remains a consideration for future privacy-preserving research. The interpretability of the generated auxiliary labels is identified as an open question for future investigation.