UniverSR: Unified and Versatile Audio Super-Resolution via Vocoder-Free Flow Matching

Technical Summary of "UNIVERSR: UNIFIED AND VERSATILE AUDIO SUPER-RESOLUTION VIA VOCODER-FREE FLOW MATCHING"

Problem Statement
Audio super-resolution (SR), or bandwidth extension (BWE), aims to reconstruct high-resolution (HR) audio signals from low-resolution (LR) counterparts by estimating missing high-frequency content. While recent advances have utilized generative models, existing approaches face significant limitations. One-stage methods (end-to-end) often suffer from training instability (GANs) or inference inefficiency (diffusion models). Two-stage pipelines, which first upsample a mel-spectrogram and then synthesize waveforms using a pre-trained neural vocoder, introduce a fundamental bottleneck: the final audio quality is constrained by the vocoder's ability to reconstruct phase information, which is omitted in mel-spectrograms. Furthermore, these pipelines often require additional post-processing and rely on separate, large-scale vocoder components.

Methodology: UniverSR
The authors propose UniverSR, a unified, vocoder-free framework that employs flow matching to directly generate complex-valued spectral coefficients. The core methodology operates as follows:

1. Spectral Inpainting Formulation: The task is framed as a spectrum inpainting problem. Given an LR signal, the model estimates the missing high-band spectrum conditioned on the low-band spectrum.
2. Flow Matching in the Spectral Domain: Instead of predicting waveforms or mel-spectrograms, the model learns the conditional distribution of complex-valued spectral coefficients. It utilizes a Vector Field Estimator (VFE) based on a U-Net architecture with ConvNeXt V2 blocks.
3. Direct Waveform Reconstruction: The model generates the high-frequency spectrum, which is concatenated with the original low-band spectrum. The full-band complex spectrum is then converted directly into the HR waveform using the inverse Short-Time Fourier Transform (iSTFT). This eliminates the need for a separate neural vocoder.
4. Conditioning Mechanism: The VFE is conditioned on a rich set of features, including:
   • Acoustic features: Extracted from the low-band spectrum via a dedicated feature encoder.
   • Positional embeddings: Sinusoidal embeddings for frequency bins to provide spectral location awareness.
   • Global context: Embeddings for time and sampling rate to handle diverse upsampling factors (×2 to ×6).
5. Training and Inference: The model is trained to minimize the difference between the predicted vector field and the target vector field derived from a conditional probability path. During inference, an Ordinary Differential Equation (ODE) solver is used to traverse the flow from noise to the target spectrum. The framework supports Classifier-Free Guidance (CFG) to balance perceptual richness and source fidelity.

Key Contributions

• Vocoder-Free Architecture: The paper introduces a novel end-to-end framework that directly reconstructs waveforms via iSTFT, removing the dependency on pre-trained neural vocoders and the associated phase reconstruction bottlenecks.
• Efficiency via Flow Matching: By utilizing flow matching, the model achieves high audio quality with substantially fewer sampling steps compared to conventional diffusion-based approaches.
• Unified Versatility: The model is trained on a diverse corpus (speech, music, and sound effects) and demonstrates state-of-the-art performance across multiple upsampling factors (from 8/12/16/24 kHz to 48 kHz) without requiring domain-specific retraining.

Experimental Results
The authors evaluated UniverSR against state-of-the-art baselines, including AudioSR and FlashSR (two-stage vocoder-based) and various single-stage diffusion models.

• Objective Metrics: On diverse datasets (Speech, Music, Sound Effects), UniverSR consistently outperformed baselines in the 2f-model (perceptual quality estimator) and LSD-HF (Log Spectral Distance in high frequencies) metrics. Notably, it achieved superior results in music and sound effect domains across all sampling rates.
• Parameter Efficiency: Despite outperforming models with ~600M parameters (due to separate diffusion and vocoder components), UniverSR achieves these results with only 57M parameters.
• Subjective Evaluation (MOS): In listening tests, the proposed model received the highest Mean Opinion Scores (MOS), surpassing even the "Ground Truth (vocoded)" baseline. The authors attribute this to the avoidance of pitch instabilities often introduced by vocoders when reconstructing harmonic-rich signals.
• Qualitative Analysis: Spectrogram visualizations confirmed that UniverSR generates cleaner, more structured high-frequency components compared to the smeared details often seen in vocoder-based outputs.

Significance and Claims
The paper claims that UniverSR addresses the critical bottleneck of two-stage pipelines where audio quality is fundamentally limited by vocoder performance. By directly modeling the conditional distribution of complex spectral coefficients, the framework simplifies the optimization process and achieves state-of-the-art fidelity. The authors emphasize that their approach offers a robust, unified solution for audio super-resolution that generalizes effectively across speech, music, and environmental sounds, proving that high-quality waveform generation can be achieved without the architectural complexity and limitations of separate vocoders.