Towards Trustworthy Audio Deepfake Detection: A Systematic Framework for Diagnosing and Mitigating Gender Bias

Technical Summary: Towards Trustworthy Audio Deepfake Detection

Problem Statement
Audio deepfake detection systems are increasingly deployed in high-stakes security applications, yet their fairness across demographic groups remains critically underexamined. While prior work has measured gender disparity in detection performance, it has failed to systematically investigate the specific sources of this bias or how to address them. Existing interventions are often applied ad hoc without confirming which bias sources are active. For instance, standard fairness assumptions suggest that reweighting imbalanced training data is a solution; however, this paper posits that in the context of audio deepfakes, observed disparities may not stem from data imbalance but from acoustic representation differences, gender leakage in learned features, and structural evaluation asymmetry.

Methodology: A Diagnosis-First Framework
The authors propose a two-stage framework designed to identify bias sources before applying targeted mitigation. The framework is evaluated on two distinct architectures: AASIST (an end-to-end spectro-temporal graph attention network) and Wav2Vec2-large+ResNet18 (a self-supervised front-end with a convolutional back-end), using the ASVSpoof5 dataset.

Stage 1: Bias Source Diagnosis
The authors conduct an eight-check diagnosis across three levels:

1. Data Level: Chi-squared tests are used to verify gender imbalance in training and evaluation sets, and to analyze the distribution of attack types.
2. Model Level: The authors analyze score distribution differences between genders, measure "gender leakage" (the ability of a classifier to predict gender from model embeddings), and use SHAP (SHapley Additive exPlanations) to determine if gender information is concentrated in specific features or diffused across many. t-SNE visualizations are used to inspect feature space separation.
3. Decision Level: The study examines training objective bias (using binary cross-entropy) and measures the gap between optimal decision thresholds for different genders.

Stage 2: Mitigation Strategies
Based on the diagnosis, the authors evaluate a comprehensive set of mitigation strategies, including three novel methods:

• Pre-processing: Sample Reweighting (S1).
• In-processing:
  • Fairness-Aware Loss (S2): Adds a penalty for FPR/FNR differences between genders.
  • Adversarial Debiasing (S3): Uses a Gradient Reversal Layer (GRL) to remove gender information from embeddings.
  • Epoch-Accumulated Fairness Regularisation (EAFR): A novel method that accumulates predictions across an entire epoch to compute fairness penalties, addressing the noise issues associated with small batch sizes.
• Post-processing:
  • Threshold Calibration (TC): Applies separate decision thresholds for each gender.
  • SHAP-Guided Feature Suppression (SGFS): A novel method that zeroes out specific dimensions identified by SHAP as gender-sensitive.
  • Gender-Neutral Embedding Alignment (GNEA): A novel method that replaces gender-sensitive feature values with the average of male and female values to reduce bias while preserving embedding scale.
• Combined Pipelines: Integrating in-processing methods with post-processing threshold calibration.

Key Results
The diagnosis revealed that bias does not stem from training data imbalance (which was balanced) but from:

1. Evaluation Protocol Asymmetry: The evaluation set had a significant gender imbalance in the ratio of bona fide to spoofed samples and attack types.
2. Score Distribution Shift: Models assigned different confidence scores to male and female spoof samples.
3. Gender Leakage: Embeddings contained gender information, though the nature of this leakage varied (concentrated in AASIST vs. diffuse in Wav2Vec2).
4. Threshold Bias: A single decision threshold disadvantaged female speakers.

Regarding mitigation:

• Sample Reweighting (S1): Actively degraded performance because the training data was already balanced, confirming that mitigation must be source-specific.
• Threshold Calibration (TC): The most reliable method, reducing false positive rate differences (dFPR) by 54–75% with no cost to detection accuracy (EER).
• Adversarial Debiasing (S3): Succeeded only for Model 1 (AASIST) where gender leakage was localized. It failed for Model 2, where leakage was diffuse, a failure correctly predicted by the Stage 1 diagnosis.
• EAFR: Outperformed standard per-batch fairness loss approaches.
• SGFS and GNEA: Effective for Model 1 but had little effect on Model 2 due to the diffuse nature of the bias.
• Combined Approaches: Combining in-processing methods with TC yielded the best overall fairness improvements.

Significance and Claims
The paper claims to present the first diagnosis-first framework for audio deepfake detection that identifies bias sources before applying fixes. Its primary contributions are:

1. Demonstrating that bias arises from multiple independent sources (evaluation asymmetry, score shifts, leakage, threshold bias) rather than just data imbalance.
2. Showing that applying generic mitigation strategies without diagnosis can harm performance.
3. Proving that simple post-processing (Threshold Calibration) can significantly reduce unfairness without retraining.
4. Introducing novel methods (EAFR, SGFS, GNEA) and validating that their efficacy depends on the specific nature of the bias (localized vs. diffuse).

The authors conclude that a "diagnosis-first" approach is necessary to avoid harmful interventions and that future benchmark designs must address structural protocol asymmetries to truly close the fairness gap. The study is limited to the ASVSpoof5 dataset and binary gender labels.