Technical Summary: PoDAR (Power-Disentangled Audio Representation)

Problem Statement

The performance of latent diffusion models for audio generation is governed by two primary factors: the expressivity of the generator and the modelability of the underlying latent space. While recent efforts have focused on improving generator architectures and the reconstruction fidelity of audio codecs, the authors argue that a representation with high reconstruction accuracy is not necessarily easy to model.

In current latent spaces, semantic content (linguistic information) is often entangled with independent nuisance variables, specifically signal power (loudness). When a downstream generator must learn a complex joint distribution over these unrelated factors, latent modelability is impaired, leading to slower convergence and suboptimal generation quality. Existing methods to improve modelability, such as representation alignment with pre-trained encoders, often require extensive ablation studies, increase memory footprints, and do not guarantee improved generative performance.

Methodology

The authors propose PoDAR (Power-Disentangled Audio Representation), a self-supervised framework designed to partition the autoencoder latent space into a power subspace and a power-invariant semantic subspace. The method consists of two main components:

1. Power Augmentation and Consistency Constraint

During autoencoder training, the framework enforces the decoupling of signal power from semantic content through:

• Randomized Power Augmentation: An augmentation function $A(\cdot)$ applies a random global gain $g$ (uniformly sampled between -6 dB and +6 dB) to the input waveform $x$, creating an augmented signal $\tilde{x} = gx$. This preserves semantic content while varying power.
• Latent Consistency Objective: The encoder output $z$ is split into power channels $z_p$ (the first $k$ channels) and content channels $z_c$ (the remaining $L-k$ channels). A consistency loss is applied to penalize deviations between the content channels of the original signal and the augmented signal:
  $$L_{PoDAR} = \|\phi(E(x))_c - \phi(E(\tilde{x}))_c\|_2^2$$
  where $\phi$ extracts the mean for VAEs or the raw output for deterministic autoencoders. This forces the encoder to route power variations exclusively into $z_p$ while keeping $z_c$ invariant.

2. Partial Classifier-Free Guidance (CFG)

The authors introduce a modified CFG rule that applies guidance exclusively to the power-invariant semantic subspace during generation. In a standard entangled space, boosting conditional adherence via CFG can inadvertently amplify nuisance variables like power, distorting the signal. In PoDAR, the guidance vector $v$ is constructed as:
$$v = [v_{cond, p}; v_{0, c} + w(v_{cond, c} - v_{0, c})]$$
where $v_{cond, p}$ and $v_{0, p}$ are the unconditional and conditional predictions for the power channels, and the guidance term is applied only to the content channels ($c$). This allows for robust steering at higher guidance scales without distorting the signal's power dynamics.

Key Contributions

1. PoDAR Framework: A self-supervised approach utilizing randomized power augmentation and a latent consistency objective to explicitly decouple signal power from semantic content in the latent space.
2. Enhanced Modelability: The explicit factorization results in a latent distribution that is significantly easier for downstream diffusion or flow models to learn, accelerating convergence by approximately 2× while achieving superior final generation quality.
3. Partial CFG Strategy: A novel guidance mechanism that restricts steering to the semantic subspace, improving robustness at higher guidance scales by preventing the amplification of power-related artifacts.

Experimental Results

The authors evaluated PoDAR using Stable Audio 1.0 (VAE1) and Stable Audio 2.0 (VAE2) codecs, paired with an F5-TTS generator, on the LibriSpeech-PC and Seed-TTS datasets.

• Convergence Speed: Generators trained on PoDAR representations achieved baseline-level speaker similarity performance in approximately half the training steps (2× acceleration).
• Final Performance:
  • Speaker Similarity: PoDAR improved speaker similarity by 0.055 for VAE1 across both LibriSpeech and Seed-TTS benchmarks compared to baselines.
  • Speech Quality (UTMOS): VAE1-PoDAR showed a 0.22 increase in UTMOS on LibriSpeech. Improvements were more moderate for VAE2 and on the Seed-TTS dataset.
  • Word Error Rate (WER): WER remained largely unchanged, as both configurations fell within the confidence intervals of the ground truth WER.
• Reconstruction vs. Generation Trade-off: While the PoDAR codecs exhibited slightly lower reconstruction fidelity (measured by ViSQOL) compared to baselines—particularly at higher disentanglement weights ($\lambda_{PoDAR}$)—this reduction was justified by the significant gains in downstream generative performance.
• Swap Test: A "swap test" confirmed that power variations were effectively isolated within the designated $k$ channels. For VAE2, a weight of $\lambda_{PoDAR}=0.1$ kept the swap gain below the 1 dB just-noticeable-difference threshold, whereas VAE1 required $\lambda_{PoDAR}=0.5$ to achieve similar isolation, albeit with a greater drop in reconstruction fidelity.

Significance and Claims

The paper claims that latent modelability is a critical, often overlooked bottleneck in audio generation. By explicitly disentangling signal power—a nuisance variable irrelevant to linguistic content—from semantic content, PoDAR creates a latent space that is "easier to model."

The authors emphasize that this approach does not aim to produce interpretable factors for human analysis but rather to optimize the statistical properties of the latent distribution for the downstream generator. The primary significance lies in the 2× acceleration in convergence and the simultaneous improvement in speaker similarity and speech quality, demonstrating that sacrificing some reconstruction fidelity for better factor disentanglement yields superior generative outcomes. Furthermore, the partial CFG strategy offers a practical solution to the instability often encountered when using high guidance scales in entangled latent spaces.

Limitations: The authors note increased computational overhead during autoencoder training due to the requirement for dual encoder passes (original and augmented inputs). Additionally, the framework has currently been validated only within the speech domain for power disentanglement, not for broader acoustic factors or other modalities.