Technical Summary: RIVET – Robust Idempotent Voice Attribute Editing

1. Problem Statement

Voice attribute editing models aim to modify specific characteristics of speech (e.g., age, gender, accent) while preserving the speaker's identity and linguistic content. While recent generative models have advanced this field through conditional synthesis and disentangled representations, they critically depend on the quality of attribute labels used during training.

In large-scale speech datasets, attribute annotations are frequently noisy, inconsistent, flipped, or automatically inferred rather than manually verified. When deep generative models are trained with such noisy supervision, they may learn incorrect associations between attributes and acoustic patterns. This leads to:

• Degraded conditional generation quality.
• Memorization of spurious correlations.
• Unstable editing behavior, where repeated or reversed edits cause "identity drift" (the speaker's identity changes) or entanglement between the speaker identity and the target attribute.

Current methods often lack mechanisms to mitigate the impact of these noisy labels, resulting in unstable mappings from input attributes to edited outputs.

2. Methodology: RIVET Framework

The authors propose RIVET (Robust Idempotent Voice Attribute Editing), a training framework designed to improve robustness to label noise by enforcing idempotency.

The Core Concept: Idempotency

An operator $f$ is idempotent if repeated application does not change the result: $f(f(x)) = f(x)$. In the context of voice editing, this implies that if an attribute is edited to a target state, applying the edit again should not alter the speech further. Conversely, if an edited speech is reverted to its original state, the representation should return to the original unedited form. This property acts as an implicit regularizer, reducing sensitivity to mislabeled examples by minimizing "drift" during repeated generation.

Architecture

RIVET builds upon a conditional voice editing architecture similar to VoiceShop but introduces a specific training strategy. The system comprises three main components:

1. Speaker Encoder: An ECAPA-TDNN model that extracts a fixed-length speaker embedding ($e$) representing identity.
2. Conditional Normalizing Flow: A flow-based model that transforms the speaker embedding based on target attribute conditions (e.g., age, gender) to produce an edited latent representation ($z_g$).
3. Speech Generator: A VITS-based model that synthesizes the final speech signal using the edited latent representation and the original speaker embedding as conditioning.

Training Objective

Unlike prior approaches that may train editing modules and speaker embeddings separately, RIVET trains all components jointly in an end-to-end framework. The total loss function ($L_{total}$) combines standard generative losses with a novel idempotency regularizer:

$$L_{total} = L_{VITS} + \lambda_f L_{flow} + \lambda_a L_{age} + \lambda_g L_{gender} + \lambda_i L_{idemp}$$

Where:

• $L_{VITS}$: Standard adversarial, feature matching, mel reconstruction, KL divergence, and duration losses.
• $L_{flow}$: Maximum likelihood objective for the normalizing flow.
• $L_{age}, L_{gender}$: Attribute classification losses for the speaker encoder.
• $L_{idemp}$: The Idempotency Loss.

Idempotency Loss Mechanism:
The framework enforces consistency in the encoded representation. After the model edits the speech and reconstructs it, the output is re-encoded. The idempotency loss penalizes the difference between the original latent representation ($z$) and the re-encoded representation ($z_{re}$) of the reconstructed speech:
$$L_{idemp} = \mathbb{E}_{x \sim p_X} [\| \text{sg}(z) - z_{re} \|_2^2]$$
Here, $\text{sg}(\cdot)$ is a stop-gradient operator. This ensures the original latent $z$ acts as a stable target, preventing trivial co-adaptation between the encoder and decoder branches while encouraging the model to map inputs to stable points on the attribute-conditioned manifold.

3. Key Contributions

1. Demonstration of Idempotency as a Regularizer: The authors show that enforcing idempotency effectively acts as a robustness regularizer for attribute-conditioned voice editing under noisy label conditions.
2. RIVET Framework: Introduction of a simple, general training framework that incorporates idempotent constraints into voice editing models without requiring architectural changes to the underlying generative backbone.
3. Open Source: The release of RIVET as the first open-source voice editing framework of its kind.

4. Experimental Results

The authors evaluated RIVET on two datasets:

• GLOBE: A large-scale dataset with naturally occurring noisy annotations.
• EARS: A high-quality dataset used to simulate controlled levels of synthetic label noise (10% to 60% label flipping).

Key Findings:

• Identity Preservation: RIVET consistently achieved higher cosine similarity between the original and reverted speech embeddings compared to the baseline, indicating better preservation of speaker identity.
• Editing Success: On the GLOBE dataset, RIVET improved attribute prediction accuracy, particularly for gender. On the EARS dataset (out-of-distribution), it maintained higher identity similarity and substantially improved gender editing accuracy under noisy conditions.
• Robustness to Noise: As label noise increased in the EARS experiments, the baseline model exhibited rapid identity drift and performance degradation. RIVET maintained higher similarity and more stable performance across all noise levels.
• Perceptual Quality: Human evaluations (Amazon Mechanical Turk) confirmed that RIVET improved perceived editing success for both age and gender attributes compared to the baseline.
• Stability: Repeated reconstruction rounds showed that the baseline suffered from identity drift, whereas RIVET maintained high similarity to the original speaker across iterations.

Both models maintained comparable naturalness (UTMOS) and intelligibility (WER) scores, indicating that the robustness gains did not come at the cost of speech quality.

5. Significance and Claims

The paper positions RIVET as a practical solution for improving the stability and robustness of voice editing systems when training data contains imperfect or noisy labels. The authors claim that:

• Idempotency serves as an effective mechanism to reduce sensitivity to mislabeled examples in conditional generative models.
• The proposed framework is model-agnostic and can be applied to attributes beyond age and gender.
• This work represents an initial step toward understanding how idempotent objectives can stabilize conditional generative models.

The authors remain modest regarding the scope, noting that while the evaluation focused on age and gender, the principle is applicable to other attributes and architectures. They identify future work as exploring these objectives in unsupervised settings and for additional attributes.