Technical Summary: Masked Energy Perturbation for Adversarial Evasion in Speaker Recognition

1. Problem Statement

The rapid advancement of deep learning in voice synthesis and deepfake technologies has introduced critical security vulnerabilities in speaker verification systems. While these systems are essential for biometric authentication in sectors like banking and healthcare, they are increasingly susceptible to adversarial attacks. Specifically, evasion attacks exploit model vulnerabilities to induce misclassification without altering the training process. Current threats include targeted noise signals capable of achieving high attack success rates (up to 95% in some contexts) and the potential for unauthorized access via voice cloning. A primary challenge in developing countermeasures is balancing attack effectiveness with perceptual quality; many existing adversarial methods introduce significant audio distortion, making them easily detectable or perceptually obvious to human listeners.

2. Methodology

The paper proposes Masked Energy Perturbation (MEP), a novel adversarial attack strategy designed to exploit speaker recognition models while minimizing perceptual distortion by adhering to psychoacoustic principles.

2.1. Small Energy Masking (SEM)

The core of the MEP approach is the selective application of perturbations to specific regions of the frequency domain.

• Signal Processing: Audio is processed using Short-Time Fourier Transform (STFT) with a Hann window (25 ms size, 12.5 ms shift) and resampled to 16 kHz.
• Energy Thresholding: The algorithm calculates the energy distribution $x[m, k]$ across time-frequency bins. A threshold $x_{th}$ is established based on the peak energy ($x_{peak}$) of each utterance. Specifically, the threshold is derived using a ratio $\eta_{th}$ (set to -20 dB in experiments) relative to the peak:
  $$x_{th} = x_{peak} 10^{\frac{\eta_{th}}{10}}$$
• Mask Generation: A binary mask $\mu[m, k]$ is generated where values are 1 if the energy is above the threshold (high-energy regions) and 0 otherwise. However, the perturbation is applied to the masked features, effectively targeting the low-energy regions (small energy regions) that are less perceptible to the human auditory model.
• Perturbation Application: The adversarial perturbation $\delta$ is multiplied by the masked feature $x_{sem}$ to generate the final masked energy perturbation $x_{mep}$. This ensures that the perturbation is concentrated in regions where the human ear is less sensitive.

2.2. Attack Strategies

The paper implements two variations of the attack based on gradient descent:

1. Basic MEP: Computes the gradient of the loss function with respect to the input, applies the energy mask to the gradient, and adds the perturbation to the original signal.
2. Iterative MEP (I-MEP): Initializes the perturbation at zero and iteratively refines it over $N$ steps (e.g., 20 iterations). At each step, the mel filterbank transformation is applied, the loss is computed, and the gradient is masked before updating the perturbation. The perturbation is clipped to maintain a constraint $\epsilon$ (set to 0.0002).

The final adversarial waveform is generated by applying an inverse STFT to the modified power spectrum.

3. Experimental Setup

• Dataset: The experiments utilized the LibriSpeech dataset (clean set), selecting an average of 113 utterances per speaker from 116 speakers.
• Target Models: Three state-of-the-art speaker encoder models were tested under white-box conditions:
  • ResNetSE34-L
  • ResNetSE34-V
  • ECAPA-TDNN
  • All models were pre-trained on the VoxCeleb dataset.
• Comparative Baselines: The proposed methods were compared against standard adversarial attack algorithms: FGSM, I-FGSM, MI-FGSM, and PGD.
• Metrics:
  • Perceptual Quality: Measured using Perceptual Evaluation of Speech Quality (PESQ) and Signal-to-Noise Ratio (SNR).
  • Attack Effectiveness: Measured using the Equal Error Rate (EER) in an Automatic Speaker Verification (ASV) task.

4. Key Results

The study demonstrates that MEP and I-MEP achieve a superior balance between evasion effectiveness and audio quality compared to conventional methods.

4.1. Audio Quality Preservation

• PESQ Scores: The I-MEP method achieved the highest PESQ scores across all models, ranging from 3.7657 to 3.7709. In contrast, traditional methods like FGSM yielded significantly lower scores (approx. 2.6).
• SNR: I-MEP maintained the highest Signal-to-Noise Ratio, ranging from 38.07 dB to 38.14 dB, indicating minimal degradation in signal integrity.
• Relative Performance: The paper notes that in PESQ evaluation, the relative performance of MEP was 26.68% higher than that of FGSM and I-FGSM.

4.2. Evasion Effectiveness

• EER Performance: The baseline EER for the models was very low (0.01% to 6.38%). Under attack, the EER increased significantly, indicating successful misclassification.
  • ResNet34-L: I-MEP achieved an EER of 44.04%.
  • ResNet34-V: I-MEP achieved an EER of 42.64%.
  • ECAPA-TDNN: I-MEP achieved an EER of 42.37%.
• Comparison: While FGSM and other methods also increased EER (ranging from ~40% to ~43%), the MEP-based methods consistently outperformed or matched the best traditional attacks while preserving significantly higher audio quality.

5. Significance and Claims

The paper claims that the Masked Energy Perturbation (MEP) approach represents a significant advancement in adversarial attacks against speaker recognition systems by leveraging psychoacoustic masking.

• Psychoacoustic Optimization: By selectively distorting low-energy regions in the mel-filterbank domain, the method exploits the limitations of the human auditory model. This allows for the injection of adversarial noise that is imperceptible to human listeners but highly effective against machine learning models.
• Privacy Protection: The authors position this method as a tool for protecting voice privacy. By generating perturbations that reduce speaker embedding similarities without degrading perceptual naturalness, MEP offers a way to obfuscate biometric identity while maintaining the utility of the voice data.
• Superiority over Elementwise Perturbation: The study concludes that MEP-based methods (specifically I-MEP) are more effective than conventional elementwise perturbation techniques (like FGSM) because they maintain high PESQ scores (>3.5) and superior SNR, making them harder to detect via audio quality analysis.

The research highlights that while deep neural networks are robust in detecting cloned voices, they remain vulnerable to carefully crafted, psychoacoustically-informed perturbations that target specific energy distributions in the frequency domain.