Technical Summary: Structural Bottlenecks on Frequency Representation in End-to-End Audio Models

Problem Statement

End-to-end neural audio models (e.g., EnCodec, DAC, Stable Audio) achieve state-of-the-art performance in compression and generation, often leading to the assumption that they directly encode interpretable features like pitch and timbre. However, interpretability studies have largely failed to recover these features from learned representations. This paper argues that strong performance can exist without direct access to interpretable time-frequency primitives. Specifically, the authors posit that strided convolutional encoders impose two structural bottlenecks that degrade access to the frequency-localized primitives (narrowband oscillations) that underlie human-meaningful audio features. These bottlenecks entangle features, making them inaccessible for independent manipulation, even if the signal information is theoretically preserved within the latent space.

Methodology

Theoretical Framework

The authors model the encoder as a mapping $E: x(t) \to z \in \mathbb{R}^d$ that compresses time-domain inputs into latents via strided convolutions. They analyze two specific failure modes:

1. Injectivity Failure (Aliasing Collapse):

   • Mechanism: Downsampling operations induce aliasing, mapping distinct input frequencies $f_i$ to the same latent frequency $f_i \mod f_s$ (where $f_s$ is the effective latent sampling rate).
   • Theory: Using the pigeonhole principle, the authors derive an analytical bound on the number of injectively representable components ($q$) based on the arithmetic relationship between signal frequencies and the stride schedule. If the number of input components $n > q$, distinct components collapse into indistinguishable proxy components.
   • Prediction: This collapse is predictable purely from architecture (stride schedule) and signal structure, independent of learned weights.

2. Separability Failure (Resolution Limit):

   • Mechanism: Even if components survive aliasing, they may remain inseparable by learned filters if the filters' bandwidths are too wide.
   • Theory: The cumulative receptive field ($R$) of the encoder sets a hard upper bound on frequency resolution ($\Delta f = f_s / R$) based on the uncertainty principle.
   • Prediction: A lightweight structural model predicts the minimum achievable bandwidth of learned filters based solely on kernel geometry (sizes, dilations, strides) without accessing weights. The authors hypothesize that learned filters operate significantly above this theoretical limit.

Empirical Validation

• Datasets: Controlled synthetic signals composed of narrowband components with known center frequencies were used to measure ground-truth collapse. Experiments covered 643 signal configurations across three state-of-the-art models: EnCodec, DAC, and Stable Audio.
• Metrics:
  • Collapse Rate: The fraction of input components mapping to shared latent proxies.
  • Separability Ratio: The ratio of component spacing to learned filter bandwidth ($\rho = \Delta c / B$).
  • Filter Bandwidth: Measured via spectral analysis of the final convolutional layer's frequency response.

Intervention: Gabor Latent Refactorization (GLRF)

To address separability (but not injectivity), the authors propose GLRF, a lightweight, post-hoc intervention that does not require retraining the encoder:

1. Decomposition: Encoder latents are passed through a fixed Gabor filterbank, parameterized to match the theoretical resolution bound ($\Delta f = f_s / R$). This re-expresses the latents in a frequency-localized basis.
2. Reconstruction: A linear mapping $M$ is learned via closed-form ridge regression to reconstruct the original latent from the Gabor-decomposed representation ($z \approx M\tilde{z}$).
3. Control: This transformation allows for targeted manipulation of specific frequency bins (e.g., substituting one pitch for another) by operating on the localized Gabor components.

Key Results

1. Predictive Accuracy of Structural Bottlenecks

• Injectivity: The analytically predicted collapse rates correlated with observed rates at $r \approx 0.99$ across all models and signal configurations.
  • Observed collapse rates ranged from 31% to 35% for realistic signal settings.
  • The study confirmed that highly composite sampling rates (e.g., 48 kHz) induce significantly more collapse than rates with lower factor complexity (e.g., 22.05 kHz or 44.1 kHz).
• Separability: Learned filters operated 9–35 times above the theoretical receptive field resolution bound.
  • For example, DAC's best filter bandwidth was 35× the theoretical limit, while EnCodec's was 10×.
  • This confirms that while stacking convolutions improves resolution over single-kernel bounds, the encoders fail to utilize the full resolution capacity permitted by their receptive fields.

2. Effectiveness of GLRF

• Resolution Recovery: GLRF reduced filter bandwidths from 10–35× the theoretical bound down to 1.5–3× the bound.
• Fidelity: Reconstruction fidelity was preserved, with latent cosine similarity $\ge 0.97$ and low log-mel error.
• Controllability:
  • Targeted Substitution: In the original latent space, targeted frequency substitution succeeded in only 30% of cases (below chance due to context contamination). In the GLRF space, success reached 100%.
  • Localization: In the Gabor space, 80% of the energy delta for a frequency substitution concentrated in just 3 bins, whereas in the original latent, the change was distributed across ~349 channels with no identifiable structure.
  • Stability: Substitution directions in the Gabor space were consistent across mixture contexts (mean cosine similarity 0.88), whereas original latent directions varied significantly (0.61).

Significance and Claims

The paper claims that the inability to control pitch and timbre in current end-to-end audio models is not merely a training deficiency but a predictable structural consequence of strided convolutional architectures.

• Architectural Determinism: The failure to access frequency primitives is determined by the stride schedule and kernel geometry, allowing these limitations to be analyzed and predicted before training.
• Entanglement vs. Absence: The models do not "forget" frequency structure; rather, they preserve it in an entangled, inaccessible form. The features exist but are not exposed in a basis that allows independent manipulation.
• Post-Hoc Recovery: The paper demonstrates that significant representational control can be recovered without retraining. GLRF serves as a proof-of-concept that re-expressing latents in a frequency-localized basis can "unlock" the signal structure already present in the encoder's parameter space.
• Design Implications: The authors suggest that stride schedule design is a critical lever for representational capacity. Choosing sampling rates with low factor complexity can substantially reduce aliasing collapse, a decision that should be made at the architectural design stage rather than relying on learned anti-aliasing.

The work positions itself within the broader field of mechanistic interpretability, arguing that for audio models, the "representational budget" provided by existing architectures is often underutilized for human-meaningful structure due to these specific geometric constraints.