Discrete Optimal Transport and Voice Conversion

This paper proposes a vector-based voice conversion method using discrete optimal transport and barycentric projection to achieve high-quality speaker alignment, while also demonstrating its potential as a novel adversarial attack that can cause synthetic speech to be misclassified as real.

The Gist

Imagine you have a voice recorder. You record your friend reading a story, but you want that recording to sound like it was spoken by a famous actor, while keeping the exact words and the rhythm of the story intact. This is Voice Conversion.

This paper is about a new, smarter way to do that conversion using a mathematical concept called Optimal Transport. Here is the breakdown in simple terms:

1. The Problem: The "Translation" Gap

Think of your voice and the actor's voice as two different languages.

• The Old Way (Averaging): Previous methods tried to translate your voice by looking at the actor's voice, finding the 4 closest-sounding "clips" (neighbors), and just averaging them together.
  • Analogy: Imagine trying to guess the flavor of a new soup by taking 4 random spoons of ingredients from a giant pot and stirring them together. You get a soup, but it might be muddy or lose the specific taste you wanted.
• The New Way (Optimal Transport): The authors treat this like a logistics problem. Imagine you have a fleet of trucks (your voice clips) and a set of warehouses (the actor's voice clips). You need to move the cargo from the trucks to the warehouses in the most efficient way possible to minimize "travel cost."
  • Instead of just averaging, this method calculates the perfect map of which truck goes to which warehouse. It creates a smooth, precise path to transform your voice into the actor's voice without losing the "soul" of the original words.

2. The Secret Sauce: "Barycentric Projection"

The paper introduces a specific trick called Barycentric Projection.

• The Analogy: Think of the actor's voice as a group of people standing in a circle.
  • The old method (Averaging) asks: "Who are the 4 people closest to you? Stand in the middle of them."
  • The new method (Barycentric Projection) asks: "Look at the 4 people closest to you, but weigh their positions based on how perfectly they match your specific voice."
  • It's like a weighted vote. If one of the 4 people is a 99% match and another is only a 60% match, the new method listens much more to the 99% match. This results in a much clearer, higher-quality voice conversion.

3. The Experiment: Does it work?

The authors tested this on a massive library of speech (LibriSpeech).

• The Results: They found that their new method (OT-BAR) consistently sounded more natural and kept the words clearer (lower "Word Error Rate") than the old averaging methods.
• The "Secret" Discovery: They found that the length of the target voice matters a lot. If you try to convert your voice to sound like an actor who only has 1 second of audio, it sounds robotic. But if the actor has minutes of audio to learn from, the conversion is magical.

4. The "Superpower" (and the Warning)

The most surprising part of the paper is a side effect they discovered.

• The Scenario: They took fake, computer-generated speech (spoofed audio) and ran it through their "Optimal Transport" filter to make it sound like real human speech.
• The Result: The filter was so good at making the fake audio sound "real" that a sophisticated security system (designed to catch fakes) got fooled! It thought the fake audio was genuine human speech over 80% of the time.
• The Metaphor: It's like a master forger who doesn't just copy a painting; they copy the texture of the canvas, the age of the paint, and the lighting so perfectly that even an art expert can't tell it's a fake.
• Why this matters: This is a double-edged sword. It proves their method is incredibly powerful at bridging gaps between different types of audio, but it also shows a new way hackers could bypass voice security systems.

Summary

This paper presents a new "GPS" for voice conversion. Instead of taking a rough, average guess at how to change a voice, it calculates the most efficient, precise route to transform one voice into another. It produces higher-quality results and, accidentally (or perhaps intentionally), creates a tool so good at mimicking reality that it can trick security systems designed to catch fakes.

Technical Summary

1. Problem Statement

The paper addresses Voice Conversion (VC), the task of transforming a speech signal from a source speaker to sound like a target speaker while preserving the original linguistic content.

• Context: Traditional deep learning VC methods often rely on spectrograms, Generative Adversarial Networks (GANs), or simple $k$-Nearest Neighbors (kNN) averaging of embeddings.
• Limitations: Recent vector-based approaches (using models like WavLM) have shown promise but often rely on simple averaging of the top-$k$ nearest neighbors in the target domain. Previous work fixed $k=4$ without rigorous ablation studies. Furthermore, these methods struggle with domain mismatches (e.g., converting synthetic speech to real speech).
• Goal: To improve VC quality by replacing simple averaging with Discrete Optimal Transport (OT) and barycentric projection, and to investigate the robustness of this approach across different data durations and domain shifts.

2. Methodology

The proposed framework operates on vector-based audio embeddings rather than raw waveforms or spectrograms.

A. Audio Representation

• Model: Uses WavLM Large (a self-supervised speech model) to extract 1024-dimensional vector embeddings every 25 ms (hop size 20 ms).
• Rationale: WavLM is trained for speaker identification, ensuring embeddings capture speaker identity effectively.

B. Discrete Optimal Transport (OT) Formulation

• Setup: Let $X$ be source embeddings and $Y$ be target embeddings. The goal is to find a transport plan $\gamma$ that minimizes the cost of moving mass from $X$ to $Y$.
• Cost Function: Instead of standard $\ell_2$ distance, the authors use cosine similarity ($c(x, y) = 1 - \cos(x, y)$) as the cost function, which is more appropriate for high-dimensional embeddings.
• Algorithm: The transport plan is computed using Entropic OT via the Sinkhorn algorithm.
• Mapping Strategies:
  1. OT-AVE (Averaging): Similar to previous work, this averages the top-$k$ target vectors based on the OT plan weights.
  2. OT-BAR (Barycentric Projection): The paper's primary innovation. Instead of a simple average, it computes a weighted sum of the top-$k$ target vectors using normalized OT weights ($\tilde{\gamma}$).
     $$T(x_i) = \sum_{j=1}^{k} \tilde{\gamma}_{ij} y_{ot(i)}^j$$
     This is interpreted as a conditional expectation $E[y|x]$. The authors restrict the sum to the top-$k$ terms to avoid noise from silence or low-energy segments, rather than using all $N$ target vectors.

C. Reconstruction

• The transformed embeddings ($\hat{y}$) are converted back into audio waveforms using the HiFi-GAN vocoder.

3. Key Contributions

1. Barycentric Projection for VC: The authors propose using barycentric projection (OT-BAR) instead of simple averaging (OT-AVE) or kNN averaging. This allows for a more nuanced mapping that respects the probability mass distribution derived from the OT plan.
2. Ablation Study on $k$: Unlike previous works that fixed $k=4$, this paper conducts a comprehensive ablation study on the number of neighbors ($k$), testing values from 1 to 40.
3. Adversarial Attack Discovery: The paper demonstrates that applying discrete OT as a post-processing step can successfully convert synthetic (spoofed) speech into the domain of real speech. This causes state-of-the-art spoof detection models (AASIST) to misclassify synthetic audio as bona fide (real) in the majority of cases.
4. Duration Analysis: The study rigorously analyzes how the duration of source and target utterances impacts conversion quality, confirming that target duration is a critical factor.

4. Experimental Results

A. Voice Conversion on LibriSpeech

Experiments were conducted on the LibriSpeech train-clean-100 dataset using an "any-to-any" conversion setup.

• Metrics: Word Error Rate (WER), Mean Opinion Score (MOS), and Fréchet Audio Distance (FAD).
• Findings:
  • OT-BAR vs. OT-AVE: OT-BAR consistently outperformed OT-AVE and KNN-VC across most $k$ values, particularly in MOS and FAD.
  • Impact of $k$: The optimal $k$ varies by dataset size, but OT-BAR remains effective even at higher $k$ values (up to 40), whereas averaging methods degrade.
  • Duration Sensitivity:
    • Target Duration: Longer target utterances significantly improved MOS and WER.
    • Source Duration: Had a lesser impact compared to target duration.
    • Asymmetric Cases: Converting short sources to long targets yielded the best MOS, while long sources to short targets yielded the lowest WER.

B. Domain Adaptation & Adversarial Attack (ASVspoof 2019)

The authors tested the method on the ASVspoof 2019 dataset to convert "fake" (synthetic) audio to "bona fide" (real) audio.

• Setup: 1000 fake recordings were converted to the voice of 1000 real speakers using the OT-BAR method.
• Detection: The converted audio was fed into the AASIST spoof detection model.
• Result:
  • Standard vocoding (without OT) did not fool the detector.
  • OT-BAR conversion caused >80% of the fake audio to be misclassified as real.
  • This highlights a severe vulnerability in current spoof detection systems and demonstrates the powerful domain-alignment capability of discrete OT.

5. Significance and Conclusion

• Technical Advancement: The paper establishes that barycentric projection is a superior method for mapping embeddings in voice conversion compared to simple averaging, offering better preservation of speaker characteristics and linguistic content.
• Security Implications: The discovery that discrete OT can effectively bridge the gap between synthetic and real speech domains poses a significant security risk. It suggests that current anti-spoofing measures may be insufficient against OT-based domain adaptation attacks.
• Practical Guidelines: The study provides clear guidelines for practitioners:
  • Use barycentric projection over averaging.
  • Ensure sufficient target speaker data (longer duration) for high-quality conversion.
  • Be aware that increasing $k$ beyond the traditional value of 4 can improve results when using barycentric projection.

In summary, this work advances the state-of-the-art in vector-based voice conversion by leveraging optimal transport theory and simultaneously uncovers a critical vulnerability in audio deepfake detection systems.