Technical Summary: SAME (Semantically-Aligned Music AutoEncoder)

Problem Statement

Modern generative media models predominantly operate on latent distributions rather than raw data. In the audio domain, Neural Audio Codecs (NACs) serve as the bridge between raw waveforms and these latent spaces. While the dominant paradigm (e.g., SoundStream, EnCodec, DAC) utilizes Vector Quantized (VQ) bottlenecks with convolutional backbones, continuous latent models (required for diffusion and flow-matching) typically rely on Variational Autoencoders (VAEs).

Existing continuous NACs face a trade-off: achieving high temporal compression ratios (to reduce the computational cost of downstream generative modeling) often degrades reconstruction quality or generative tractability. Furthermore, standard VAE formulations can struggle with the specific statistical requirements of high-compression audio representation. The authors identify a need for an architecture that achieves extreme temporal compression (4096×) while maintaining high reconstruction fidelity, preserving stereo imaging, and ensuring the latent space is well-aligned for downstream generative tasks.

Methodology

The proposed SAME (Semantically-Aligned Music autoEncoder) is a stereo music and general audio autoencoder that synthesizes transformer-based backbones with specific semantic regularization strategies.

1. Architecture

The model follows an encoder-bottleneck-decoder structure with three core components:

• Patching Pretransform: Stereo waveforms $(B, 2, T)$ are partitioned into non-overlapping patches of $P=256$ samples per channel, reshaping the input to $(B, 2P, T/P)$. This achieves a 256× temporal downsampling without learned parameters.
• Transformer Resampling Blocks (TRB): Instead of strided convolutions, SAME uses a query-based transformer resampling block to achieve the remaining compression.
  • Mechanism: The TRB interleaves input embeddings with learnable output embeddings. In encoder mode, it downsamples by a stride $S$; in decoder mode, it upsamples.
  • Attention Strategy: To handle variable-length audio and ensure linear complexity, the model employs sliding-window attention (preferred for performance) or chunked attention with a midpoint shift (for CPU deployment to avoid boundary artifacts).
  • Normalization: The architecture utilizes Dynamic Tanh (DyT) instead of LayerNorm/RMSNorm to avoid issues with silence or low-level noise. The final decoder layers use sinusoidal activations to better reconstruct waveform-level details.
• Soft-Normalization Bottleneck: Rather than a standard VAE bottleneck, SAME employs a lightly constrained bottleneck. It uses a learnable per-channel affine transform followed by division by a running standard deviation (tracked via exponential moving average). A KL-like regularisation loss encourages zero-mean, unit-variance statistics along both time and channel axes, preventing drift and outliers.

2. Training Objectives

The training regimen combines reconstruction, adversarial, and auxiliary losses to shape the latent space:

• Spectral Reconstruction Losses: A multi-resolution STFT loss (7 resolutions) comprising:
  • Spectral Contrast: A symmetric, scale-invariant loss replacing standard spectral convergence.
  • Adaptive Log-Magnitude: An L1 loss on log-magnitudes where the normalization factor $\sigma$ is adaptive to signal statistics, preventing gradient dominance by noise floors.
  • Phase-Derivative Losses: Instead of raw phase differences, the model operates on normalized complex phasors to measure Instantaneous Frequency (IF) and Group Delay (GD) coherence, preserving transient fidelity and stereo imaging.
• Adversarial Training: A relativistic paired GAN objective using a multi-view ensemble of discriminators:
  • Convolutional Discriminator: Extends EnCodec with multi-scale STFT, PQMF filter-bank, and chroma views.
  • Transformer Discriminator: Replaces STFT/chroma discriminators with TRB-based versions and adds patched-waveform discriminators to avoid harmonic aliasing.
• Auxiliary Losses (Semantic Alignment):
  • Generative Alignment (Flow-Matching): A small unconditional diffusion transformer is trained jointly on the latent space to shape the geometry for downstream flow-matching generation.
  • Semantic Regression: Lightweight linear regressors predict perceptual features (chroma and interaural level difference) directly from the latent, enforcing semantic structure.
  • Contrastive Latent Alignment: A transformer-based critic aligns the latent representation with audio features (wavelet decomposition) and text embeddings (T5Gemma) using a contrastive loss.

3. Model Variants

Two configurations are released:

• SAME-L: An 852M-parameter model with 12 transformer blocks (dim 1536), optimized for high quality.
• SAME-S: A distilled 108M-parameter model (6 blocks, dim 768) using chunked attention, designed for CPU deployment on edge devices. SAME-S is distilled from a frozen SAME-L teacher.

Key Results

The paper evaluates SAME against recent open-weights continuous-latent autoencoders (Stable Audio Open, $\epsilon$ar-VAE, CoDiCodec, ACE-Step).

• Compression and Speed: SAME achieves a 4096× temporal compression ratio (double the standard for audio autoencoders). In terms of Real-Time Factor (RTF), SAME-S is 6–7× faster than convolutional VAE baselines, and SAME-L is 2× faster despite having significantly more parameters.
• Reconstruction Quality:
  • Objective: SAME-L achieves state-of-the-art performance on MELlog1p (multi-resolution log-mel error) and competitive results on SI-SDR and STFTlog1p compared to $\epsilon$ar-VAE.
  • Subjective: In a MUSHRA listening test, SAME-L received the highest rating (82.2 ± 16.6), outperforming $\epsilon$ar-VAE (77.6) and other baselines.
• Generative Tractability (Ablation Study): An ablation study demonstrates that the auxiliary losses are critical. A configuration with the Soft-Normalization bottleneck but without auxiliary losses performed worse than a standard VAE baseline. Adding flow-matching alignment and semantic regression allowed the high-compression (4096×) model to outperform lower-compression (1024×) VAEs on generative metrics (FAD-CLAP and MuQ-Eval).

Significance and Claims

The authors claim that SAME represents a significant advancement in audio autoencoding by demonstrating that high temporal compression does not necessitate a trade-off in quality or generative utility, provided the latent space is properly regularized.

Specifically, the paper posits that:

1. Semantic Regularization is Key: The combination of flow-matching alignment, semantic regression, and cross-modal contrastive alignment allows a simpler, non-VAE bottleneck to outperform traditional VAEs at high compression ratios.
2. Transformer Primitives are Efficient: The reliance on well-optimized transformer primitives (TRB) combined with high compression yields substantial computational benefits, making high-quality audio generation more accessible, particularly for CPU-based edge deployment via the SAME-S variant.
3. Phase and Stereo Preservation: The specific phase-aware losses and stereo-specific training objectives successfully maintain transient fidelity and stereo imaging even at extreme compression levels.

The work concludes that SAME provides a robust, open-weights foundation for next-generation generative audio models, balancing reconstruction fidelity, generative performance, and inference speed.