Gender Fairness in Audio Deepfake Detection: Performance and Disparity Analysis

This paper analyzes gender bias in audio deepfake detection using the ASVspoof 5 dataset and a ResNet-18 classifier, demonstrating that while aggregate metrics like Equal Error Rate may suggest low disparity, fairness-aware evaluation reveals significant gender-specific error distributions that necessitate more equitable and robust detection systems.

The Gist

Here is an explanation of the paper, translated into everyday language with some creative analogies.

The Big Picture: The "Voice Impersonator" Problem

Imagine a world where a thief can use a magic wand to perfectly copy your voice. They could call your bank, say "Transfer all my money," and sound exactly like you. This is what Audio Deepfakes are: AI-generated voices that sound real but aren't.

For years, scientists have been building "Voice Detectors" (like a security guard) to spot these fakes. The goal is simple: Is this voice real, or is it a robot?

But here's the catch: The researchers in this paper asked a question nobody had really checked before: "Does this security guard treat men and women fairly?"

The Experiment: A Test Drive for Different "Ears"

The researchers didn't just build one detector; they tested five different "ears" (ways of listening to the sound) to see if they were biased.

1. The Baseline (AASIST): The current champion, a high-tech AI model.
2. Four New "Ears" (Features):
   • LogSpec & CQT: These are like listening to the shape of the sound waves (visualizing the sound).
   • WavLM & Wav2Vec: These are like "super-listeners" trained on millions of hours of human speech to understand context.

They used a massive dataset called ASVspoof5, which is like a giant library of real and fake voices, carefully balanced with an equal number of men and women.

The Twist: The "Average" Lie

Usually, when we test a security system, we just look at the overall score.

• Analogy: Imagine a teacher grading a class. If the class average is 85%, the teacher says, "Great job!"

But what if the boys averaged 95% and the girls averaged 75%? The average is still 85%, but the system is failing the girls.

This paper argues that looking only at the average score (called EER) is dangerous. It hides the fact that the system might be failing specific groups of people.

The Findings: Who Got the Short End of the Stick?

The researchers used five specific fairness tests (like a report card with five different subjects) to see how the models treated men vs. women. Here is what they found:

1. The "Champion" (AASIST) is actually the fairest.
   Even though it made more mistakes on women than men, the gap was tiny. It was the most balanced of the bunch.

2. The "Visualizers" (CQT) were the most biased.
   This model was terrible at fairness. It was like a security guard who was very strict with women but very lenient with men. It made huge mistakes on one group compared to the other.

3. The "Super-Listeners" (WavLM) were good at accuracy but had a bias.
   WavLM was the best at catching fakes overall, but it still had a slight preference. It was slightly better at spotting fakes made to sound like men, meaning women's voices were sometimes harder to verify.

4. The "Predictive Parity" Surprise:
   Almost every model was slightly better at being "right" when it said a voice was a man's voice, compared to a woman's. This means if the system says "That's a fake," it's more likely to be correct if the voice sounds male.

The Big Lesson: Accuracy $\neq$ Fairness

The most important takeaway from this paper is this: A system can be highly accurate overall but still be unfair.

• Analogy: Imagine a metal detector at an airport. If it beeps 99% of the time when a man walks through, but only beeps 90% of the time when a woman walks through (even if she has a weapon), the system is "accurate" overall, but it is unsafe for women.

The paper proves that if we only look at the "Total Score," we miss these dangerous gaps.

Why Does This Matter?

If we build voice security systems for banks, phones, or courts without checking for gender bias:

• Women might be locked out of their own bank accounts because the system thinks their voice is a fake.
• Men might get away with fraud because the system is too quick to believe them.

The Conclusion: What's Next?

The authors didn't invent a new "magic fix" in this paper. Instead, they sounded an alarm. They are saying:

"Stop just checking if the model is smart. Start checking if the model is fair."

They suggest that in the future, we need to design AI that doesn't just learn to spot fakes, but learns to spot fakes equally well for everyone, regardless of whether they sound like a deep bass or a high soprano.

In short: We need to make sure our AI security guards don't have a blind spot for half the population.

Technical Summary

Here is a detailed technical summary of the paper "Gender Fairness in Audio Deepfake Detection: Performance and Disparity Analysis."

1. Problem Statement

The rapid advancement of AI-generated audio (deepfakes) poses significant security risks, including identity theft and misinformation. While Audio Deepfake Detection (ADD) models have improved in overall accuracy, there is a critical gap in understanding gender bias within these systems.

• The Core Issue: Standard evaluation metrics (like Equal Error Rate) often mask performance disparities between demographic groups.
• The Hypothesis: Due to natural physiological differences in pitch, vocal range, and speaking patterns between male and female speakers, ADD models may exhibit systematic bias, leading to unequal error rates (e.g., higher false positives for one gender) even when overall accuracy appears balanced.
• Research Gap: Most existing fairness studies focus on image/video domains. Audio deepfake detection lacks formal fairness evaluations using established demographic parity metrics.

2. Methodology

A. Dataset

• Source: The ASVspoof 5 dataset, the latest benchmark for spoofing-aware speaker verification.
• Characteristics: Unlike previous editions, ASVspoof 5 offers a near-balanced distribution of male and female speakers across training, development, and evaluation splits.
• Preprocessing: Audio signals were converted to mono, resampled to 16 kHz, and standardized to 4.0 seconds (zero-padded or truncated).

B. Feature Representations

The study evaluated four distinct feature extraction methods to determine if specific features introduce bias:

1. Log-Spectrogram (LogSpec): Time-frequency representation capturing energy variations.
2. Constant-Q Transform (CQT): Emphasizes pitch and harmonic structures.
3. WavLM: A self-supervised deep learning embedding trained on large-scale corpora.
4. Wav2Vec 2.0: Another self-supervised model learning contextual representations from raw audio.

C. Model Architecture & Training

• Classifier: A unified ResNet-18 architecture was used for all feature sets to ensure a fair comparison.
• Baseline: The state-of-the-art AASIST model (end-to-end, raw audio input) was used as a comparative baseline.
• Training Protocol:
  • Optimizer: AdamW.
  • Loss Function: Class-weighted cross-entropy to handle class imbalance.
  • Regularization: Early stopping and learning rate scheduling (ReduceLROnPlateau).
  • Evaluation: Models were tested on the official ASVspoof 5 evaluation set, partitioned into Female-only, Male-only, and Combined subsets.

D. Fairness Evaluation Framework

Beyond the standard Equal Error Rate (EER), the study employed five established fairness metrics to quantify disparities:

1. Statistical Parity (SP): Difference in positive prediction rates between genders.
2. Equal Opportunity (EOP): Difference in True Positive Rates (TPR).
3. Equality of Odds (EO): Difference in False Positive Rates (FPR).
4. Predictive Parity (PP): Difference in Precision (Positive Predictive Value).
5. Treatment Equality (TE): Ratio of False Positives to False Negatives across groups.

Statistical significance was determined using two-proportion z-tests with Holm-Bonferroni correction ($\alpha = 0.05$).

3. Key Results

A. Fairness Analysis (Disparities)

• AASIST (Baseline): Showed the smallest overall disparities across all five metrics, making it the most balanced system despite a consistent slight bias favoring males.
• LogSpec: Demonstrated the fairest behavior among feature-based models, with minimal gaps in Statistical Parity, Equal Opportunity, and Equality of Odds.
• CQT: Exhibited the largest cumulative disparity, particularly in Treatment Equality ($\Delta = 1.216$), showing a severe bias favoring females.
• Self-Supervised Models (WavLM vs. Wav2Vec): Both showed female-favoring bias, but WavLM was significantly fairer than Wav2Vec, especially in Equality of Odds and Treatment Equality.
• Predictive Parity: All systems showed a male-favoring trend in precision, suggesting a dataset or score-distribution effect rather than a feature-specific one.
• Statistical Significance: Almost all observed disparities were statistically significant ($p < 0.05$), confirming they are systematic rather than random noise.

B. Performance Analysis (EER)

• Best Performance: WavLM achieved the lowest overall EER (22.00%), followed closely by AASIST (23.26%).
• Worst Performance: CQT had the highest EER (~43%), indicating poor discriminative capability.
• Gender Gap in EER:
  • AASIST: Showed a notable gap where females had higher error rates (24.92%) than males (21.37%).
  • WavLM: Showed the smallest gap (22.28% vs. 21.65%).
  • Crucial Insight: A model can have a low EER (high accuracy) but still exhibit significant fairness issues. Conversely, a model with a slightly higher EER (like LogSpec) might be more equitable.

4. Key Contributions

1. Systematic Fairness Audit: First comprehensive analysis of gender bias in audio deepfake detection using the ASVspoof 5 dataset and a unified ResNet-18 classifier.
2. Metric Expansion: Demonstrated that relying solely on EER is insufficient; incorporating five fairness metrics reveals hidden demographic failure modes.
3. Feature Sensitivity: Identified that feature choice (e.g., CQT vs. LogSpec) drastically impacts fairness, with some features introducing severe gender bias while others remain balanced.
4. Baseline Comparison: Validated that the state-of-the-art AASIST model, while highly accurate, still exhibits measurable gender bias, necessitating fairness-aware improvements.

5. Significance and Conclusion

• Reliability of Metrics: The study proves that aggregate metrics like EER are unreliable indicators of system equity. A system can be "accurate" overall while failing specific demographic groups.
• Equitable Deployment: For biometric systems to be trustworthy, they must perform consistently across all user groups. The findings highlight that current models may disproportionately harm one gender (e.g., higher false positives for males in some configurations, or higher false negatives for females in others).
• Future Directions: The authors conclude that future research must move beyond accuracy optimization to include fairness-aware loss design, subgroup reweighting, and adversarial debiasing. Understanding the source of these biases (whether in the model architecture or the feature extraction process) is critical for developing robust, equitable deepfake detection systems.