DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

1. Problem Statement

Voice anonymization aims to conceal speaker identity while preserving linguistic content and para-linguistic information (such as emotion and emphasis). However, a fundamental utility–privacy trade-off exists:

• Privacy: Speaker identity is tightly coupled with prosodic patterns (pitch, rhythm, intonation). Preserving these patterns improves expressiveness but risks identity leakage to speaker recognition systems.
• Utility: Discarding prosody to ensure privacy degrades the naturalness, emotional content, and meaning of the speech.

Existing methods typically operate at fixed design points:

• ASR-TTS systems: Prioritize privacy by discarding all source prosody, leading to robotic or flat speech.
• Voice Conversion (VC) systems: Attempt to preserve prosody but often suffer from imperfect disentanglement, leaving residual identity leakage.
• Perturbation methods: Randomly alter prosodic features, causing unpredictable degradation in utility.

The Gap: There is a lack of frameworks that allow for continuous, inference-time control over the trade-off between prosody preservation (utility) and anonymization strength (privacy) within a single model.

2. Methodology: DiffAnon

The authors propose DiffAnon, a diffusion-based voice anonymization framework that utilizes Classifier-Free Guidance (CFG) to enable explicit control over prosody preservation.

Core Architecture

• Base Model: A Denoising Diffusion Probabilistic Model (DDPM) inspired by NaturalSpeech2, utilizing 40 WaveNet-style residual blocks.
• Semantic Prior (Content): The model uses SpeechTokenizer, a Residual Vector Quantization (RVQ) codec.
  • The first-level quantization embeddings ($Q_1$) are treated as a speaker-agnostic semantic prior representing linguistic content.
  • The diffusion process refines the remaining acoustic details ($Q_{2:8}$) on top of these semantic embeddings.
• Conditions:
  1. Content ($c_{sem}$): $Q_1$ embeddings (fixed, added directly to the input).
  2. Prosody ($c_{pro}$): Frame-level latent features ($z_{mpm}$) extracted from a Masked Prosody Model (MPM), capturing pitch, energy, and voicing without speaker identity.
  3. Speaker ($c_{spk}$): An utterance-level embedding from a pre-trained FreeVC encoder, replaced with a pseudo-speaker embedding ($\psi$) during inference to anonymize identity.

Inference Strategy: Classifier-Free Guidance (CFG)

The key innovation is the use of CFG to interpolate between different levels of prosody preservation without retraining.

• Training: The model is trained with random condition dropping (50% full, 30% prosody dropped, 20% prosody + speaker dropped) to enable unconditional generation capabilities.
• Prosody-Adjusted Guidance: During inference, the output is a weighted combination of:
  1. A prediction conditioned on content and pseudo-speaker only (no source prosody).
  2. A prediction conditioned on content, pseudo-speaker, and source prosody.
  • Formula: $\hat{x}_{CFG} = \hat{x}_{uncond} + w_{pro}(\hat{x}_{cond} - \hat{x}_{uncond})$
  • Control: The weight $w_{pro}$ acts as a continuous knob.
    • $w_{pro} = 1$: Full preservation of source prosody (High Utility, Lower Privacy).
    • $w_{pro} = 0$: Complete removal of source prosody (High Privacy, Lower Utility).
• Pseudo-Speaker Guidance: A secondary CFG mode ($w_{spk}$) can be used to strengthen the anonymization by aggressively guiding the generation away from the source speaker identity.

3. Key Contributions

1. First Controllable Framework: DiffAnon is the first voice anonymization system to provide structured, interpolatable, inference-time control over prosody preservation.
2. Novel Formulation: It introduces a diffusion paradigm that refines acoustic details over speaker-agnostic semantic embeddings (RVQ space), naturally aligning with the iterative refinement of speech codecs.
3. Unified Trade-off Navigation: It demonstrates that a single trained model can navigate the entire utility–privacy spectrum simply by adjusting the CFG guidance weight, eliminating the need for multiple specialized models.
4. Open Source: The code and pretrained models are publicly released.

4. Experimental Results

Experiments were conducted following the VoicePrivacy Challenge 2024 protocol using the LibriTTS dataset.

• Controllability: By varying $w_{pro}$ from 1.0 to 0.0, the system achieved a smooth, monotonic transition:
  • Prosody Fidelity: F0 correlation dropped from 76.67% ($w_{pro}=1$) to 64.09% ($w_{pro}=0$).
  • Emotion Recognition (UAR): Dropped from 52.32% to 47.38%.
  • Privacy (EER): The Equal Error Rate (resistance to speaker verification) improved (increased) from 33.09% to 42.43% (Lazy-informed attacker on Libri-test).
• Utility Preservation:
  • Word Error Rate (WER): Remained stable and competitive (4.62% to 5.61%) across all settings, demonstrating that content preservation is robust even when prosody is suppressed.
  • Comparison: DiffAnon outperformed or matched several top-performing baselines (T8, T9, T10) while offering the unique ability to adjust the trade-off dynamically.
• Privacy: The strongest anonymization setting (null prosody + pseudo-speaker guidance) achieved an EER of 48.16%, competitive with the strongest fixed-design baselines.

5. Significance

• Paradigm Shift: DiffAnon moves voice anonymization from a "fixed design" problem to a "controllable generation" problem. It proves that prosody is the primary driver of the utility–privacy trade-off and that this trade-off can be mathematically controlled via CFG.
• Practical Application: This allows developers to tune anonymization systems based on specific use cases (e.g., high privacy for legal testimony vs. high expressiveness for audiobooks) using a single model.
• Future Direction: The work establishes a foundation for extending controllable anonymization to other prosodic factors like duration and rhythm, further refining the balance between privacy and naturalness.