Technical Summary: PHALAR - Phasors for Learned Musical Audio Representations

1. Problem Statement

Current audio representation learning largely adopts paradigms from computer vision, treating spectrograms as static 2D images. A cornerstone of these architectures is the use of pooling operations, such as Global Average Pooling (GAP), to enforce translational invariance. While beneficial for semantic classification (e.g., identifying instrument presence), this invariance is detrimental for tasks requiring structural coherence, such as music mixing and stem separation.

The specific challenge addressed is stem retrieval: given a partial audio submix (e.g., drums and bass), the model must identify which missing stems temporally and harmonically fit with it. Existing foundation models (e.g., CLAP, CDPAM) and even specialized coherence models (e.g., COCOLA) rely on GAP or real-valued global pooling. These operations discard temporal ordering and phase information, collapsing distinct rhythmic alignments into identical latent representations. Consequently, these models are "structurally blind," failing to distinguish between a coherent mix and a misaligned one, even if the instruments are identical.

2. Methodology

The paper proposes PHALAR (Phasors for Learned Musical Audio Representations), a contrastive learning framework that shifts the inductive bias from temporal/phase invariance to equivariance.

Core Architectural Shift

The method leverages the Fourier Shift Theorem, which posits that a temporal translation in the input domain corresponds to a phase rotation in the frequency domain. PHALAR explicitly models coherence by shifting the representation space from real-valued magnitudes to complex-valued phasors.

The architecture consists of three primary components:

1. Harmonic Backbone (Real-Valued):

   • A lightweight 2D CNN processes Constant-Q Transform (CQT) spectrograms. CQT is used because its logarithmic spacing ensures pitch shifts are purely linear translations, allowing kernels to recognize harmonic intervals (e.g., major thirds) identically across all keys.
   • The backbone employs an axial residual design to decouple spectral and temporal processing:
     • Frequency-wise Convolutions (3×1): Extract harmonic relationships within timesteps.
     • Time-wise Convolutions (1×3): Capture temporal evolution.
     • Point-wise Convolutions (1×1): Facilitate feature mixing.
   • Strided time-wise convolutions compress the temporal dimension by a factor of 32× before pooling.

2. Learned Spectral Pooling (The Bridge):

   • Replacing standard GAP, this layer projects the feature maps into the complex frequency domain.
   • The process involves:
     1. Projecting flattened features onto a learned basis ($W_{proj}$) to encode harmonic interval structure and absolute frequency position.
     2. Applying a Real Fast Fourier Transform (RFFT) along the temporal axis.
   • Result: Temporal shifts are converted into phase rotations. The magnitude $|S_{c,d}|$ encodes the prevalence of a harmonic pattern, while the phase $\angle S_{c,d}$ explicitly encodes its temporal shift.

3. Complex-Valued Projection Head (CVNN):

   • A Complex-Valued Neural Network processes the spectral representation. Standard real-valued MLPs would destroy phase structure; thus, PHALAR uses a CVNN (Trabelsi et al., 2018) where every operation is phase-equivariant ($f(x \cdot e^{i\theta}) = f(x) \cdot e^{i\theta}$).
   • The head consists of complex linear layers, Complex RMSNorm, and a phase-preserving modReLU.
   • Similarity Metric: To quantify structural coherence, the model uses a Phase-Aware Bilinear Similarity:
     $$s(z_x, z_y) = \Re(z_x^H W z_y)$$
     where $W$ is a learnable complex weight matrix. This allows the model to apply learnable phase rotations to "align" stems, accounting for micro-timing deviations (e.g., a "laid back" groove). During inference, a symmetric score is used to ensure commutativity.

3. Key Contributions

• Novel Framework: Introduction of PHALAR, a contrastive framework that explicitly decouples harmonic content from rhythmic alignment using complex-valued phasors.
• State-of-the-Art Performance: Achieves a relative accuracy increase of up to ≈70% over the previous state-of-the-art (COCOLA) on stem-to-mix retrieval tasks.
• Efficiency: Requires < 50% of the parameters of the baseline (2.3M vs. 5.2M) and offers a 7× training speedup (50 vs. 340 GPU-hours), driven by the parameter-efficient axial backbone and the elimination of CPU-bound pre-processing.
• Orthogonality of Similarity and Coherence: Demonstrates that semantic similarity models (CLAP, CDPAM) perform at random chance on coherence tasks, while PHALAR correlates significantly with human perception.
• Emergent Capabilities: Without explicit supervision, PHALAR captures robust musical structures, enabling zero-shot beat tracking and linear chord probing.

4. Experimental Results

The framework was evaluated on MoisesDB, Slakh2100, and ChocoChorales datasets.

• Retrieval Accuracy: PHALAR established new SOTA across all datasets. On MoisesDB with $K=64$ (a difficult setting with high tonal collision probability), PHALAR achieved 70.87% accuracy compared to COCOLA's 41.84% and MERT's 45.85%.
• Human-Centric Validation: In a subjective listening test with 22 participants, PHALAR's embedding distances showed the highest correlation with human coherence ratings (Pearson $\rho = 0.387$, Spearman $r_s = 0.414$), significantly outperforming all baselines including COCOLA and Audiobox metrics.
• System-Level Evaluation: When evaluating generated stems, PHALAR's aggregated scores reproduced human ranking orders for different generative models, whereas standard set-level metrics like Fréchet Audio Distance (FAD) failed to align with human judgment regarding conditional coherence.
• Ablation Studies:
  • Removing phase equivariance (using magnitude only) caused a 10.3% accuracy drop.
  • Using Global Average Pooling instead of Learned Spectral Pooling caused an 18.9% drop.
  • Replacing CQT with Mel-spectrograms resulted in a 1.66% drop, confirming the necessity of strict log-spacing for pitch equivariance.
• Emergent Structures:
  • Beat Tracking: PHALAR achieved an F1-score of 0.627 on beat tracking without any rhythmic supervision, recovering tempo via geometric interference patterns.
  • Chord Probing: A linear probe on frozen PHALAR embeddings achieved 55.2% accuracy on chord classification, outperforming the Chroma CQT baseline (50.6%).

5. Significance and Claims

The paper claims that PHALAR represents a fundamental shift in audio representation learning, moving from "structurally blind" semantic models to "phase-aware" coherence models.

• Geometric Inductive Bias: The work posits that music, being fundamentally periodic, is the ideal modality for geometric biases. By modeling temporal shifts as phase rotations, PHALAR resolves rhythmic coherence where magnitude-based models fail.
• Metric for Generative Audio: PHALAR provides a robust, reference-aware metric for evaluating generative audio, addressing a critical gap where standard metrics (like FAD) assess marginal distribution realism but fail to assess conditional structural fit.
• Broader Impact: The authors suggest that the principles of complex-valued metric learning and phase preservation extend beyond audio to other phase-sensitive domains such as Radar systems, Medical Imaging (MRI), and Time Series Analysis.

Limitations: The authors acknowledge that PHALAR relies on explicit geometric priors (RFFT) assuming temporal periodicity. Performance degrades with non-periodic tempo changes (e.g., rubato), arrhythmic content, or heavily compressed audio formats. Additionally, the model's training on Western popular music may bias its notion of "coherence" against stylistic micro-timing deviations in other genres.