R&B -- Rhythm and Brain: Cross-subject Decoding of Music from Human Brain Activity

Technical Summary: R&B - Rhythm and Brain: Cross-subject Decoding of Music from Human Brain Activity

Problem Statement
The study addresses the challenge of decoding musical stimuli from human brain activity measured via functional magnetic resonance imaging (fMRI). While previous research has explored brain decoding for various stimuli, music presents specific difficulties due to the low temporal resolution and signal-to-noise ratio (SNR) of fMRI data, as well as the inherent variability in brain anatomy and function across different subjects. Existing approaches often rely on subject-specific models or proprietary computational models, limiting generalizability. The authors aim to develop a cross-subject decoding pipeline that leverages open-source, pre-trained multimodal models to map neural activity to latent representations of music, thereby enabling the retrieval of musical stimuli from brain data.

Methodology
The study utilizes the GTZan fMRI dataset, comprising data from five participants who listened to 540 musical stimuli across 10 genres. The methodology consists of two primary pipelines: encoding and decoding.

1. Preprocessing and Alignment: fMRI data were preprocessed using FSL and Nilearn, including motion correction, co-registration to MNI space, and detrending. To address intersubject variability, three alignment strategies were tested:

   • Anatomical Alignment: Mapping data to a standard anatomical space.
   • Functional Alignment (Hyperalignment): Using Procrustes analysis to align functional responses across subjects.
   • Linear Alignment: Employing ridge regression to aggregate cross-subject brain data into a unified model, addressing multicollinearity in high-dimensional fMRI data.

2. Feature Extraction: The authors used the Contrastive Language-Audio Pretraining (CLAP) model to extract latent representations of the musical stimuli. CLAP, a multimodal neural network, projects audio features into a shared latent space, providing compact representations that the authors hypothesize share topological similarities with brain representations.

3. Encoding Models: Voxel-wise encoding models were constructed to map CLAP latent representations to brain activity. A cross-validation scheme measured the correlation between predicted and actual brain activity for each voxel. An empirical threshold of 0.1 correlation was applied to identify Regions of Interest (ROIs) responsive to musical stimuli. This process identified 833 voxels, primarily located in lateral and temporal regions.

4. Decoding Pipeline: A retrieval-based decoding model was trained using ridge regression to map brain activity from the identified ROIs to CLAP features. For each test sample, the model predicted CLAP features and retrieved the top-k (k=5) closest musical stimuli from the latent space based on Euclidean distance.

Key Contributions

• Cross-Subject Decoding Framework: The paper introduces a streamlined pipeline for cross-subject music decoding that integrates functional and anatomical alignment techniques, moving beyond subject-specific models.
• Use of Open-Source Foundation Models: The study leverages the open-source CLAP model for feature extraction, contrasting with prior works that relied on proprietary models like MuLAN or MusicLM.
• Identification of Music-Responsive ROIs: By using voxel-wise encoding models, the authors delineated specific brain regions (including the superior temporal gyrus, primary auditory cortex, planum temporale, and inferior parietal lobule) that are most responsive to musical stimuli.
• Performance Benchmarking: The study provides a comparative analysis of different alignment techniques and embedding models, demonstrating the superiority of functional alignment methods in this context.

Results

• Identification Accuracy: The proposed methods achieved state-of-the-art identification accuracy. The linear alignment method yielded the highest performance with an identification accuracy of $0.9012 \pm 0.01573$, followed by hyperalignment at $0.8805 \pm 0.0231$. These results significantly outperformed baselines using SoundStream ($0.674$), w2v-BERT ($0.837$), and MuLan embeddings ($0.817–0.876$), as well as the authors' own anatomical alignment approach ($0.7746$).
• Genre Classification: The confusion matrix indicated high accuracy for classical, jazz, and pop genres. Misclassifications were more frequent for disco and metal, likely due to overlapping acoustic features in the CLAP latent space.
• Temporal Dynamics: Analysis of decoding accuracy over time showed an increasing trend in identification accuracy as the stimulus duration progressed, peaking in the later part of the 18-second window, suggesting that prolonged neural engagement enhances decoding performance.

Significance and Implications
The authors claim that their findings demonstrate the feasibility and high accuracy of decoding music from cross-subject neural activity. They suggest that these results pave the way for neural-based music retrieval systems, which could enable personalized music recommendations and therapeutic applications. Specifically, the ability to decode music from brain activity could inform music therapy protocols by tailoring interventions to individual neural responses. The paper also posits that this work lays the groundwork for future explorations into the therapeutic potential of music in neurological settings and the development of brain-computer interfaces. The authors note that while the current study focuses on retrieval due to fMRI's low temporal resolution, future work could extend these techniques to generative music systems using higher-resolution neuroimaging methods like EEG or iEEG.