Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

This paper reveals that hidden states in end-to-end full-duplex speech models like SALM-Duplex and Moshi significantly leak speaker identity, and proposes two streaming anonymization methods using Stream-Voice-Anon that effectively mitigate this privacy risk while maintaining low-latency dialogue performance.

The Gist

Imagine you are talking to a super-smart, always-listening AI assistant. Unlike old chatbots that wait for you to finish speaking before they reply, this new type of AI (called Full-Duplex) listens and talks at the same time, just like a real human conversation. It never stops "thinking" about what you say, keeping a running mental note of your voice, your tone, and your style.

The paper you shared is a privacy detective story about these new AI assistants. Here is the breakdown in simple terms:

🕵️‍♂️ The Problem: The AI Knows Who You Are

The researchers asked a scary question: Even if the AI doesn't record your voice, does its "brain" (the hidden data it processes) still remember who you are?

They tested two popular AI models, SALM-Duplex and Moshi.

• The Discovery: Yes, the AI's brain is leaking your identity. Even though the AI is just trying to understand your words, its internal "notes" are so detailed that a hacker could use them to identify exactly who you are, just like a fingerprint.
• The Analogy: Imagine you are whispering a secret to a friend. You think you are safe because you aren't shouting. But this AI is like a friend who, while listening, is also secretly writing down the exact pitch of your voice, the rhythm of your breathing, and the specific way you say "hello." If someone steals those notes, they know it's you, even if they never heard your actual voice.

🛡️ The Solution: The "Voice Mask"

The researchers didn't just find the problem; they built two different "masks" to hide your identity while still letting the AI understand you. They call these Anonymization Setups.

1. The "Voice Changer" (Anon-W2W)

• How it works: Before your voice even reaches the AI's brain, it passes through a special filter that changes your voice into a different voice (like a voice changer app).
• The Analogy: It's like putting on a disguise before walking into a room. The AI hears a stranger's voice, so it can't tell it's you.
• The Catch: The AI still has to "re-read" this disguised voice to understand it, which takes a tiny bit of extra time.

2. The "Secret Code" (Anon-W2F)

• How it works: This is the smarter, faster version. Instead of changing the sound waves, it converts your voice directly into a secret code (numbers) that the AI understands, but which has your identity stripped out.
• The Analogy: Instead of wearing a mask, you are speaking in a secret language that only the AI understands, but the language itself has no clues about who you are.
• The Result: This method was the winner. It made it 3.5 times harder for a hacker to identify you compared to the original system. It got so good that the hacker was basically just guessing (like flipping a coin), which is perfect privacy.

⚖️ The Trade-off: Privacy vs. Performance

Every time you put on a mask, you might lose a little bit of clarity.

• Privacy: The "Secret Code" method made the AI almost impossible to hack for identity.
• Quality: The AI's answers became slightly less perfect (a small drop in how human-like the text or speech sounded), but the researchers say this is a small price to pay to stop people from stealing your identity.
• Speed: The system is still fast enough to have a real-time conversation, though the "Voice Changer" method was a bit slower than the "Secret Code" method.

📉 The "Time" Factor

The researchers also found that the longer you talk, the more the AI remembers about you.

• Without a mask: After just a few sentences, the AI's internal notes are so full of your identity that a hacker could easily find you.
• With a mask: Even after a long conversation, the AI's notes remain "blurry" regarding your identity, keeping you safe.

🏁 The Bottom Line

This paper is a wake-up call. As we move toward AI assistants that are always listening and talking, we can't just assume they are private. The "brain" of these AIs naturally stores your identity like a fingerprint.

The good news? The researchers showed us how to build privacy shields (masks and secret codes) that let us talk to these AIs safely, ensuring that while the AI knows what we are saying, it never knows who we are.

Technical Summary

1. Problem Statement

End-to-end (E2E) full-duplex speech dialogue systems (e.g., SALM-Duplex, Moshi) represent a shift from turn-taking to "always-on" simultaneous listening and speaking. These systems route raw user audio continuously through a Large Language Model (LLM) backbone, maintaining persistent hidden states throughout the conversation.

The Core Issue: While these models excel at dialogue utility, their hidden representations likely encode speaker identity (voice characteristics, speaking style). This creates a significant privacy risk:

• Re-identification: Even if the conversation content is anonymized, the hidden states may allow an attacker to identify who is speaking.
• Regulatory Risk: Under regulations like GDPR, the mere presence of identifiable speaker information in model representations constitutes a compliance risk.
• Gap in Research: While speaker identity leakage in static speech encoders (e.g., wav2vec 2.0) is well-documented, no prior work has examined whether the continuous, always-on hidden states of full-duplex LLMs retain sufficient identity information to enable re-identification.

2. Methodology

A. Systems Analyzed

The authors evaluated two prominent E2E full-duplex models:

1. SALM-Duplex: Uses an ASR-initialized continuous encoder to process user audio.
2. Moshi: Uses a discrete encoder (Residual Quantization Vector Quantization codec) to convert audio into discrete tokens.

B. Evaluation Protocol

• Attacker Model: A "lazy-informed" attacker scenario following the VoicePrivacy 2024 protocol. A speaker verification probe (ECAPA-TDNN) is trained on hidden states extracted from the LLM.
• Metrics:
  • Equal Error Rate (EER): Primary privacy metric. Higher EER (approaching 50%) indicates better privacy (random chance).
  • Linkability: A legally validated metric measuring the ability to link different utterances to the same speaker.
  • Utility: Measured via sBLEU, sBERT (quality), and latency metrics (FRL, RTFx).
• Granularity: Analysis was performed layer-wise (early, middle, late layers) and turn-wise (accumulation of leakage over dialogue duration).

C. Proposed Solutions: Streaming Anonymization

The authors propose two setups using Stream-Voice-Anon to mitigate leakage without breaking the real-time dialogue flow:

1. Anon-W2W (Wave-to-Wave):

   • Mechanism: Applies Stream-Voice-Anon as a pre-processing step to transform the raw input waveform into an anonymized waveform before it enters the model's original encoder.
   • Applicability: Tested on both SALM-Duplex (continuous) and Moshi (discrete).
   • Drawback: Introduces redundancy; the anonymized waveform is synthesized and then re-encoded by the model.

2. Anon-W2F (Wave-to-Feature):

   • Mechanism: Replaces the model's original encoder with a discrete encoder that natively supports anonymization (Stream-Voice-Anon operates directly on the feature/token domain).
   • Applicability: Demonstrated on SALM-Duplex (replacing its continuous encoder).
   • Advantage: Eliminates the redundant waveform synthesis step, operating natively in the feature domain.

3. Key Results

A. Baseline Leakage (No Anonymization)

• Significant Identity Leakage: Both models leak substantial speaker identity.
  • Moshi (Discrete): Extremely high leakage with an EER of 6.4% (near-perfect identification).
  • SALM-Duplex (Continuous): Moderate leakage with an EER of 28.5%.
• Layer-wise Analysis:
  • Moshi: Leakage is uniform and severe across all transformer layers.
  • SALM-Duplex: Leakage is strongest in early layers and decreases slightly in deeper layers, suggesting deeper layers abstract away some speaker features, but identity remains detectable.
• Turn-wise Analysis: Privacy degrades rapidly within the first few turns of conversation. Without anonymization, Linkability rises sharply, dropping privacy into the "low protection" zone quickly.

B. Effectiveness of Anonymization

• Anon-W2W:
  • Raised EER for Moshi from 6.4% to 36.9%.
  • Raised EER for SALM-Duplex from 28.5% to 34.6%.
  • Maintained high dialogue quality (sBERT retention of 78–93%).
• Anon-W2F (SALM-Duplex):
  • Achieved the strongest protection, raising EER from 11.2% (discrete baseline) to 41.0%.
  • This approaches the 50% random-chance ceiling, effectively anonymizing the speaker.
  • Efficiency: Faster than W2W (RTFx 2.5 vs 1.6) because it avoids redundant waveform synthesis.

C. Trade-offs

• Quality: Anonymization caused a moderate drop in dialogue quality (sBERT dropped 7–22%), but the privacy gains (EER improvement of 21–477%) were deemed to outweigh the costs.
• Latency: Adding the anonymization module reduced the Real-Time Factor (RTFx) significantly (from ~17–263x to ~1.6–2.5x), but all systems remained viable for real-time interaction (RTFx > 1).

4. Key Contributions

1. First Empirical Audit: The first study to characterize speaker identity leakage in the hidden states of E2E full-duplex dialogue LLMs, proving that "always-on" architectures inherently expose speaker identity.
2. Granular Analysis: Provided layer-wise and turn-length-wise analyses showing that leakage persists across all transformer layers and accumulates rapidly over dialogue turns.
3. Novel Architectures: Proposed and validated two streaming anonymization frameworks (Anon-W2W and Anon-W2F) that successfully mitigate leakage while maintaining sub-second response latencies.
4. Privacy-Utility Benchmark: Established a new benchmark showing that feature-domain anonymization (Anon-W2F) can achieve near-perfect privacy (EER ~41%) with acceptable utility loss.

5. Significance and Conclusion

This paper highlights a critical, previously unexamined vulnerability in the next generation of voice AI: the persistent encoding of speaker identity in LLM hidden states.

• Implication: Simply using an LLM for dialogue does not guarantee privacy; in fact, the continuous processing of audio may make re-identification easier than in traditional cascaded systems.
• Solution Path: The authors demonstrate that privacy-by-design is feasible. By integrating anonymization directly into the feature extraction pipeline (Anon-W2F), it is possible to approach the theoretical limit of anonymization (random chance) without rendering the system unusable.
• Future Work: The authors call for extending these methods to other architectures (like Moshi), improving the efficiency of anonymization modules, and evaluating against more sophisticated attacker models.

In summary, the paper argues that for E2E full-duplex systems to be compliant with privacy regulations and safe for deployment, speaker anonymization must be an integral part of the model architecture, not an optional add-on.