Music Restoration via Latent Operator Optimization and Diffusion Model Priors

Technical Summary: Music Restoration via Latent Operator Optimization and Diffusion Model Priors

Problem Definition
Music restoration aims to recover a clean, unprocessed signal from an observed recording degraded by unknown effects, distortions, or corruptions (e.g., reverberation, compression, clipping, or saturation). Existing supervised approaches rely on paired clean/distorted training data and predefined distortion families, limiting their applicability when the forward degradation process is unknown, complex, or lacks a clean reference (e.g., historical recordings). While unsupervised blind inverse problem frameworks exist, they are often restricted to specific forward-model families or operate directly in the waveform domain, where optimization can be unstable.

Methodology: LOUDAR
The authors propose LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration), a general-purpose, unsupervised framework that operates within the latent space of a pretrained audio autoencoder (AE). The method formulates restoration as a blind inverse problem in the latent domain, alternating between estimating the clean latent representation and learning the distortion operator.

1. Latent Space Formulation:
   Instead of solving the inverse problem $y = f(x) + n$ in the waveform domain, LOUDAR maps the observed distorted waveform $y$ and the clean target $x$ into latent vectors $z_y$ and $z_0$ using a frozen encoder $E$. The distortion process is modeled as a parametric latent operator $g_\phi$ and a residual term $r$:
   $$z_y \approx g_\phi(z_0) + r$$
   Here, $g_\phi$ is a learnable operator with unknown parameters $\phi$, and $r$ captures localized artifacts or noise not representable by $g_\phi$.

2. Diffusion Prior:
   An unconditional Latent Diffusion Model (LDM), trained on clean music, serves as a prior $p(z)$ over the clean latent distribution. This prior regularizes the inference by steering the estimated latent vector toward the manifold of clean recordings. The LDM utilizes a Diffusion Transformer (DiT) backbone trained with a Rectified Flow objective.

3. Distortion-Blind Inference (EM-style):
   At inference time, LOUDAR employs an Expectation-Maximization (EM) style alternating procedure along the reverse diffusion trajectory:

   • E-step: Estimate the clean latent vector $\hat{z}_0$ using the diffusion prior (denoiser) given the current state.
   • M-step: Update the latent operator parameters $\phi$ and residual $r$ to minimize the reconstruction error between the observed latent $z_y$ and the modeled distortion $g_\phi(\hat{z}_0) + r$.
   • Update: Perform a guided Ordinary Differential Equation (ODE) step using the sum of the unconditional score (from the prior) and the likelihood score (derived from the current distortion model).

4. Latent Operator Architecture:
   The operator $g_\phi$ is implemented as a causal convolutional network with low-rank constraints. It consists of bias-enabled low-rank causal Conv1D layers with residual connections. This design ensures the model is flexible enough to represent diverse distortions while remaining constrained enough to prevent overfitting during the per-input optimization.

Key Contributions
The paper identifies three primary contributions:

1. Formulation: A novel formulation of music restoration as a blind inverse problem specifically within the latent space of an autoencoder.
2. Architecture: A compact, general-purpose latent operator architecture designed to stabilize blind optimization by balancing flexibility with capacity control.
3. Evaluation: A comprehensive evaluation across singing voice and guitar domains, demonstrating that the method improves over degraded inputs and competes with supervised and unsupervised baselines.

Experimental Results
The authors evaluated LOUDAR on two tasks: singing voice effect removal/distortion restoration and guitar distortion removal (recovering Direct Injection signals from amplifier recordings).

• Singing Voice:

  • Objective Metrics: Using specialized embeddings (AFx-Rep, FxEncoder++, CLAP, Singer-ID), LOUDAR achieved competitive results. Notably, it yielded the lowest scores on Singer-ID embeddings, indicating superior preservation of speaker identity compared to other methods.
  • Subjective Listening: In MUSHRA listening tests, LOUDAR outperformed the supervised "Apollo" model (which scored best on objective metrics but worse perceptually) and the "LDM Conditional" baseline. The paper reports a statistically significant difference for the comparison between LOUDAR and the LDM conditional baseline ($p < 0.001$), while noting that Apollo was outperformed by LOUDAR without specifying statistical significance for that specific comparison.
  • Generalization: When tested on aggressive distortions (clipping, bitcrushing, GSM compression) using English samples despite a Chinese training prior, LOUDAR demonstrated more stable performance than SRS and VoiceFixer, which struggled with distortions underrepresented in their training data.

• Guitar Restoration:

  • LOUDAR achieved the strongest results on FxEncoder++ and CLAP embeddings.
  • It clearly outperformed UnAFx (an unsupervised waveform-domain baseline), supporting the hypothesis that latent space restoration is more tractable for complex distortions.
  • It performed comparably to supervised baselines (LDM Conditional, Latent Regressor) on most metrics, despite being unsupervised.

Significance and Claims
The paper claims that LOUDAR offers a broadly applicable solution for music restoration where the degradation chain is unknown. By shifting the optimization to the latent space, the method compresses both the representation and the distortion model, making the blind optimization of complex, unknown effect chains tractable.

The authors emphasize that LOUDAR is particularly effective in scenarios where supervised models struggle to generalize due to mismatched training data or overly complex effect chains. However, they modestly acknowledge limitations:

• Latent Fidelity: The output quality is bounded by the reconstruction fidelity of the underlying autoencoder.
• Prior Mismatch: The diffusion prior may introduce timbral or phonetic artifacts if the training distribution (e.g., Chinese singing) differs significantly from the test input (e.g., English singing).
• Noise Sensitivity: The method is designed to project inputs toward a clean manifold rather than explicitly separate large additive noise components, leading to performance degradation in high-noise conditions.

The work suggests that combining LOUDAR with discriminative denoising approaches could be a natural direction for future work to improve robustness against strong measurement noise.