Technical Summary: VoxGuard – Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks

1. Problem Statement

Voice anonymization systems aim to conceal speaker identity and sensitive attributes (e.g., gender, accent) while preserving speech intelligibility. However, current evaluation standards rely almost exclusively on the Equal Error Rate (EER) of Automatic Speaker Verification (ASV) systems. The authors argue that EER is insufficient for privacy assessment because it captures an average-case operating point, often obscuring "tail behavior" where adversaries can mount high-precision attacks. In privacy-critical contexts, even a small number of successful identifications at a low False Positive Rate (FPR) constitutes a meaningful breach. Furthermore, existing metrics often fail to adequately quantify the leakage of non-identity attributes like gender and accent.

2. Methodology: The VoxGuard Framework

The paper introduces VoxGuard, a framework grounded in Differential Privacy (DP) and Membership Inference Attacks (MIA) to formalize and evaluate speech privacy. The framework defines two complementary notions of privacy and evaluates them in the low-FPR regime to expose worst-case leakage.

2.1. Privacy Definitions

VoxGuard formalizes two distinct privacy goals:

• User Privacy: Prevents speaker re-identification. This is framed as a membership inference problem where an adversary determines if a probe utterance belongs to a specific enrolled user. The authors define strict (same text, different speakers) and relaxed (arbitrary text) cases, bounded by $(\epsilon, \delta)$-DP inequalities.
• Attribute Privacy: Protects sensitive traits (e.g., gender, accent). This is framed as determining if a probe utterance exhibits a specific attribute value. The framework defines strict (single-attribute change, others fixed) and relaxed (group comparison, multiple traits may vary) cases.

2.2. Evaluation Metrics

Instead of relying on EER, VoxGuard evaluates performance at low-FPR operating regions ($10^{-2}$ to $10^{-4}$). The primary metric is the worst-case privacy loss ($\hat{\epsilon}$), calculated as the log-likelihood ratio between True Positive Rate (TPR) and FPR:
$$\hat{\epsilon} = \max \left\{ \ln \left( \frac{\text{TPR}}{\text{FPR}} \right), \ln \left( \frac{1 - \text{FPR}}{1 - \text{TPR}} \right) \right\}$$
This approach prioritizes the ability of an adversary to make high-confidence, low-error identifications.

2.3. Experimental Setup

• Datasets: The study utilizes both synthetic data (generated via XTTS and Kokoro TTS using Vox-Profile attributes) and real speech data (VCTK and ParaSpeechCaps).
• Anonymization Baseline: The VoicePrivacy 2024 B5 (ASR-BN) system, which uses fundamental frequency (F0) and VQ bottleneck features with a HiFi-GAN vocoder.
• Adversary Models:
  • User Privacy: Two attacker variants were tested: a Pretrained-Informed model (off-the-shelf ECAPA-TDNN) and a Fine-tuned-Informed model (adapted on anonymized data). Scoring strategies included Average Similarity and Max Similarity (the latter modeling a worst-case attacker).
  • Attribute Privacy: Evaluated using Cosine Similarity baselines and Transparent Learned Attacks (logistic regression classifiers trained on embedding differences/concatenations).

3. Key Results

3.1. User Privacy: EER Masks Severe Risks

The results demonstrate a significant divergence between EER and low-FPR performance:

• EER Limitations: Systems with comparable EERs can differ by orders of magnitude in the low-FPR regime. For instance, in the USER-PRIVACY-RELAXED scenario, a Fine-tuned-Max similarity attacker achieved a TPR of 18.83% at an FPR of $10^{-4}$, whereas an Average similarity attacker achieved only 0.18% at the same FPR, despite both having similar EERs (~35-36%).
• Impact of Adaptation: Fine-tuned attackers consistently outperformed pretrained baselines, indicating that ignoring attacker adaptation leads to an overestimation of privacy.
• Scoring Strategy: Max-similarity scoring consistently revealed higher leakage than average-similarity, particularly with longer enrollment lengths, suggesting that averaging embeddings dilutes strong matches while max-scoring preserves worst-case risks.

3.2. Attribute Privacy: Near-Perfect Leakage

The study found that current anonymization methods fail to protect sensitive attributes:

• Simple Attacks Succeed: Transparent attacks using logistic regression on pretrained embeddings achieved near-perfect accuracy in recovering gender and accent from anonymized speech under the ATTR-PRIVACY-RELAXED setting.
• Ineffectiveness of Fine-tuning: Fine-tuning embedding models on anonymized data did not improve privacy; instead, it led to unstable evaluation outcomes, likely due to distorted embedding geometry. The underlying attributes remained separable.

4. Contributions and Significance

The paper makes four primary contributions:

1. Reframing Privacy: It reframes speech privacy as a membership inference problem, formalizing User Privacy and Attribute Privacy using DP-style indistinguishability bounds.
2. Low-FPR Evaluation: It argues that privacy must be evaluated in the low-FPR regime rather than via EER, positing that even a few reliable breaches constitute a privacy violation.
3. Exposing Metric Gaps: The results reveal that ASV models with similar EERs can expose vastly different levels of risk in the low-FPR regime, suggesting that average-case metrics substantially underestimate leakage.
4. Attribute Vulnerability: It demonstrates that speaker attributes (gender, accent) remain unprotected by current anonymization pipelines, as simple transparent attacks can fully expose these traits.

Significance:
The authors conclude that EER is a misleading metric for privacy assurance. They advocate for VoxGuard as a new benchmark for evaluating privacy leakage, emphasizing the need for worst-case analysis and explicit defenses for demographic attributes. The work highlights that current anonymization systems may provide a false sense of security by reporting stable EERs while failing to prevent high-precision identification and attribute inference.