⚡ electrical engineering

CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning

This paper introduces CultureMERT-95M, a multi-culturally adapted music foundation model trained via a novel two-stage continual pre-training strategy on diverse non-Western traditions, which significantly improves cross-cultural auto-tagging performance while maintaining proficiency on Western benchmarks and offering a viable alternative through task arithmetic.

Original authors: Angelos-Nikolaos Kanatas, Charilaos Papaioannou, Alexandros Potamianos

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Angelos-Nikolaos Kanatas, Charilaos Papaioannou, Alexandros Potamianos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Music is a universal language, yet the computers we use to understand it often speak only one dialect. For years, the most powerful artificial intelligence models designed to listen to and categorize music have been trained almost exclusively on Western styles, such as pop, rock, and classical symphonies. These systems have become incredibly good at recognizing the patterns of those specific traditions, but they struggle when faced with the vast, rich diversity of the world's other musical cultures. They miss the subtle melodic structures of Indian classical music, the unique rhythmic cycles of Turkish traditions, or the distinct scales of Greek folk songs. This limitation means that for billions of people whose musical heritage falls outside the Western canon, these digital tools remain blind, unable to preserve, recommend, or analyze the art that defines their communities.

Researchers have long sought a way to teach these models to listen more broadly without having to rebuild them from the ground up, a process that would be prohibitively expensive and time-consuming. The challenge lies in adapting a model that has already learned one set of rules to understand a completely different set of rules without causing it to forget what it already knew. This is the central problem addressed by a new study introducing a system called CultureMERT. The researchers aimed to take a foundation model that had been trained on a thousand hours of predominantly Western music and gently guide it to understand the musical languages of the Eastern Mediterranean and the Indian subcontinent, all while ensuring it did not lose its ability to recognize the Western styles it was originally built to know.

To achieve this, the team employed a strategy known as continual pre-training. Instead of discarding the existing model and starting over, they fed it a new, carefully balanced diet of audio data. This new dataset included 650 hours of music drawn from four distinct traditions: Turkish makam, Hindustani classical, Carnatic classical, and Greek traditional music. However, simply feeding this new data to the model caused it to stumble, a phenomenon where the computer's performance on its original tasks would crash as it tried to learn the new ones. To solve this, the researchers devised a two-stage approach. In the first stage, they introduced the model to a smaller portion of the new music while keeping its most complex internal layers frozen, allowing only the basic sound-processing parts to adjust. They also mixed in a small amount of Western music during this phase to remind the model of its original training. In the second stage, they unlocked the entire model and trained it on the full 650-hour dataset, allowing it to fully integrate these new cultural nuances.

The results were striking. When tested on tasks that required the model to identify instruments, moods, or specific musical modes in non-Western music, the adapted model showed a significant improvement, averaging a nearly five percent increase in accuracy compared to the original version. Crucially, this gain did not come at the cost of its previous knowledge; the model retained its ability to recognize Western music with almost no loss in performance. The study also explored a different method called task arithmetic, which involves mathematically combining the "knowledge" of several smaller models, each trained on a single culture, into one unified system. This approach performed just as well as the direct training method on non-Western tasks and even outperformed it on Western benchmarks, suggesting that merging specialized models is a powerful alternative to retraining them all at once.

The researchers found that the model's ability to transfer knowledge between cultures was not uniform. For instance, the model trained on Carnatic music from South India showed a strong ability to generalize to Hindustani music from North India and even to Turkish traditions, likely because these systems share deep theoretical roots in melody and rhythm. In contrast, the model struggled more to transfer knowledge to Greek traditional music, which, while distinct, shared fewer structural similarities with the Indian and Turkish datasets in the specific way the computer encoded sound. By analyzing the digital "tokens" the model uses to represent sound, the team discovered that the mathematical distance between these musical traditions predicted how well the model would learn them, offering a new way to plan future training data.

Ultimately, this work demonstrates that artificial intelligence can be made more inclusive without sacrificing its core capabilities. The researchers have released their adapted model, CultureMERT, to the public, providing a tool that can finally listen to the world's music with a much broader ear. They acknowledge that while their model is a significant step forward, it is not a perfect solution; the underlying technology used to convert sound into digital tokens was still trained on Western music, which may limit how deeply it can grasp certain cultural subtleties. Nevertheless, by showing that a model can learn new musical languages while remembering the old ones, this study offers a clear path toward a future where digital music systems are truly global, capable of honoring the full spectrum of human musical expression.

Technical Summary: CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning

1. Problem Statement

While recent music foundation models have significantly advanced audio representation learning, their effectiveness remains limited when applied to diverse, non-Western musical traditions. Existing models, such as MERT-v1, are predominantly trained on Western-centric datasets (e.g., pop, rock, classical), causing them to fail in capturing unique melodic structures, modal systems, and rhythmic patterns inherent in traditions like Turkish, Indian, and Greek music. This limitation restricts the applicability of these models in region-specific recommendation systems and cultural heritage preservation. Furthermore, training new models from scratch for every cultural domain is computationally prohibitive. The paper addresses the need for inclusive, culturally aware models that can generalize beyond Western traditions without catastrophic forgetting of prior knowledge.

2. Methodology

2.1 Continual Pre-Training (CPT) Strategy

The authors propose CultureMERT-95M, a foundation model adapted from MERT-v1-95M using a two-stage continual pre-training strategy. This approach aims to stabilize training and mitigate catastrophic forgetting when adapting to a shifted data distribution (from Western to multi-cultural).

Pre-training Objective: The model retains the self-supervised masked language modeling (MLM) objective of MERT, utilizing two teacher models:
1. Acoustic Teacher (EnCodec): Discretizes audio into tokens from 8 residual vector quantization (RVQ) codebooks.
2. Musical Teacher: Reconstructs Constant-Q Transform (CQT) spectrograms to encode pitch and harmonic structure.
  The total loss is a weighted sum of the acoustic token prediction loss ( $L_{RVQ}$ ) and the CQT reconstruction loss ( $L_{CQT}$ ).
Two-Stage Adaptation:
- Stage 1 (Stabilization): To address the "stability gap" and prevent performance drops, the model is trained on a smaller subset (100 hours) of multi-cultural data. Crucially, only the CNN-based feature extractor and codeword embedding layers are updated, while the Transformer encoder remains frozen. To mitigate forgetting, 20% of this data is "Western replay" (Music4All dataset). A learning rate re-warming schedule is applied.
- Stage 2 (Full Adaptation): The Transformer encoder is unfrozen, and training continues on the full 650-hour multi-cultural dataset (comprising Greek, Turkish, and Indian traditions). A less aggressive learning rate schedule is used to balance plasticity and stability.
Data Composition: The training mix includes 200 hours each from Turkish-makam, Hindustani, and Carnatic datasets, and 50 hours from the Lyra corpus (Greek traditional music), totaling 650 hours.

2.2 Task Arithmetic (TA)

As an alternative to multi-cultural CPT, the authors investigate Task Arithmetic, a weight-space merging technique.

Mechanism: Task vectors ( $\tau_i$ ) are computed as the difference between parameters of single-culture adapted models ( $\theta_i$ ) and the base pre-trained model ( $\theta_{pre}$ ).
Merging: A unified multi-cultural model ( $\theta_{merged}$ ) is constructed by algebraically summing these task vectors with scaling factors ( $\lambda_i$ ):
$\theta_{merged} = \theta_{pre} + \sum_{i=1}^{N} \lambda_i \tau_i$
This approach allows for the creation of a multi-cultural model without additional training, provided single-culture adapted models are available.

3. Key Contributions

First Application in MIR: This is the first study to explore continual pre-training and task arithmetic specifically for cross-cultural adaptation in Music Information Retrieval (MIR).
Stabilized CPT Strategy: The proposal of a two-stage CPT strategy with learning rate re-warming and staged parameter updates, designed to handle computational constraints and prevent catastrophic forgetting during domain shift.
Performance Gains: The development of CultureMERT-95M, which achieves an average improvement of 4.9% in ROC-AUC and Average Precision (AP) across diverse non-Western music auto-tagging tasks compared to the original MERT-v1.
Task Arithmetic Validation: Demonstration that task arithmetic performs on par with multi-cultural CPT on non-Western tasks and shows no regression on Western datasets, offering a training-free alternative.
Cross-Cultural Analysis: An investigation into transferability, revealing that single-culture models transfer with varying effectiveness, while the multi-culturally adapted model achieves the best overall performance.
Open Release: Public release of CultureMERT-95M and CultureMERT-TA-95M to foster research in world music representation.

4. Experimental Results

4.1 Evaluation Setup

Models were evaluated on both Western (MTAT, FMA-medium) and non-Western (Turkish-makam, Hindustani, Carnatic, Lyra) auto-tagging tasks using a probing-based evaluation (frozen backbone + shallow MLP) to assess representation quality.

4.2 Performance Metrics

Non-Western Adaptation: CultureMERT-95M outperformed the base MERT-v1 by an average of 4.9% across ROC-AUC and AP on non-Western tasks. It also surpassed previous state-of-the-art (SOTA) results on all evaluated non-Western tagging tasks.
Western Retention: The multi-cultural model exhibited minimal forgetting on Western benchmarks, with an average drop of only 0.05% in ROC-AUC and AP.
Task Arithmetic vs. CPT: The Task Arithmetic variant (CultureMERT-TA) performed comparably to CultureMERT on non-Western tasks and even surpassed both the base and multi-cultural CPT models on Western benchmarks and the Lyra dataset.
Single-Culture Transfer: Single-culture adapted models (e.g., MakamMERT) performed best on their respective in-domain tasks. However, even low-resource adaptations (e.g., LyraMERT trained on 50 hours) yielded noticeable gains on other non-Western tasks, indicating positive cross-cultural transfer.

4.3 Cross-Cultural Transferability & Token Similarity

Transfer Asymmetry: The study observed strong transfer between Turkish-makam and Carnatic music, likely due to shared modal frameworks and microtonality. The Carnatic-adapted model showed the highest transferability across other domains.
Token Similarity: Analysis using Jensen-Shannon divergence and cosine distance on EnCodec tokens revealed that non-Western traditions share significant token-level similarity, distinct from Western datasets. Greek traditional music (Lyra) aligned more closely with non-Western traditions than Western ones. These metrics correlated with transfer performance, suggesting token similarity can predict cross-cultural transferability.

5. Significance and Claims

The paper claims that CultureMERT-95M successfully bridges the gap in music foundation models by enabling effective adaptation to underrepresented musical cultures without sacrificing performance on Western benchmarks. The authors emphasize that:

Cultural Awareness is Feasible: It is possible to develop models that generalize beyond Western-centric traditions using efficient continual pre-training strategies.
Efficiency: The two-stage CPT strategy effectively balances the stability-plasticity dilemma under constrained computational resources (using a single GPU with limited batch sizes).
Alternative Pathways: Task arithmetic offers a viable, training-free alternative for creating multi-cultural models, provided single-culture adaptations exist.
Resource Availability: By releasing the models and code, the authors aim to support further research in world music representation learning and the development of more inclusive MIR systems.

The authors modestly note limitations, including the potential suboptimality of the frozen EnCodec tokenizer (pre-trained on Western music) for diverse musical languages and the need for future work to scale to additional cultures and explore fine-grained ablation studies.