🤖 machine learning

MulTTiPop: A Multitrack Transcription Dataset for Pop Music

This paper introduces MulTTiPop, a dataset comprising 572 diverse pop music segments aligned with multitrack MIDI recordings, which is used to evaluate and demonstrate significant performance gaps in state-of-the-art automatic music transcription models.

Original authors: Nathan Pruyne, Benjamin Stoler, William Chen, Chien-yu Huang, Shinji Watanabe, Chris Donahue

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nathan Pruyne, Benjamin Stoler, William Chen, Chien-yu Huang, Shinji Watanabe, Chris Donahue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to read music by ear. You want it to listen to a full pop song—drums, bass, guitars, vocals all mixed together—and write down exactly what every single instrument is playing. Sounds cool, right? But here's the problem: until now, we didn't have a good "answer key" to check if the robot was doing it right. Most existing answer keys were either for solo piano (too easy), made up by computers (too perfect), or only listed the main melody and chords (too simple).

Enter MulTTiPop, a new dataset created by researchers at Carnegie Mellon University to fix this. Think of MulTTiPop as a massive, 3.5-hour playlist of real pop hits from the 1930s all the way to the 2000s, paired with a "magic sheet music" that tells you exactly what every instrument is doing, down to the millisecond.

The Great Matching Game

How did they build this? It was like a high-stakes game of "Find the Match."

The Clues: They started with a giant list of YouTube videos of pop songs (from a database called TheoryTab) and a giant library of digital sheet music (from the Lakh MIDI Dataset).
The First Filter: They used computer magic to match song titles and artist names. If the names were almost identical, they kept the pair. This narrowed it down to about 1,164 potential matches.
The Tempo Twist: Here's where it gets tricky. Even if the song is the same, the "beat" in the digital sheet music might be slightly faster or slower than the real YouTube video. It's like two people trying to dance to the same song but one is slightly dragging their feet.
The Human Touch: To fix the timing, the researchers used a clever trick. They found a specific "anchor beat" (a moment where the music definitely lines up) and then stretched or squished the digital sheet music to match the real video's rhythm perfectly. But computers aren't perfect at picking that anchor beat. So, they asked human annotators (students who love music and tech) to listen to the video and look at the digital sheet music side-by-side to pick the correct alignment.

In the end, they successfully aligned 572 segments of music. That's about 3.5 hours of audio, covering 374 unique songs by 263 different artists.

The Reality Check: Robots Still Have a Long Way to Go

The researchers didn't just build the dataset; they immediately put the best robot musicians to the test. They asked two top-tier AI models (called MT3 and YourMT3+) to listen to these MulTTiPop clips and write down the notes.

The results? The robots struggled.

The best model only got about 38% of the note starts correct (a metric called "Onset F1").
The robots often got confused when the music got dense. They might hear the melody but miss the bass, or they'd get stuck on one instrument and ignore the rest.
One model tried to guess the vocals but labeled them as a saxophone (which is funny, but wrong).

The paper suggests that the reason these robots fail is that they were mostly trained on fake, computer-generated music (like the Slakh2100 dataset) rather than real, messy, commercial pop recordings. It's like teaching a chef to cook by only showing them perfect, plastic food models, then handing them a real, sizzling steak and expecting them to know what to do.

What This Dataset Is (and Isn't)

The authors are very clear about the rules of the game:

It IS for testing: MulTTiPop is a "final exam" for music transcription AI. It's designed to show us where the robots are failing so we can fix them.
It is NOT for training: The researchers explicitly rule out using this dataset to teach the robots. Because the songs are copyrighted commercial hits and the dataset is relatively small, using it to train a model would be unfair and legally shaky.
It has a bias: The paper admits that the songs chosen are mostly from Western pop music (American artists, Western harmony). It suggests that if a robot does well on MulTTiPop, it's good at Western pop, but we can't be sure it would do well on music from other parts of the world or different traditions.

The Bottom Line

MulTTiPop is a vital new tool that finally gives us a realistic "answer key" for complex pop music. It proves that while our AI music transcribers are getting better at simple tasks, they still have a substantial room for improvement before they can truly understand the messy, wonderful chaos of a real pop band. The robots are currently getting a "C-" on this test, and the researchers are hopeful that with this new dataset, they can help them study harder and get an "A" next time.

Technical Summary: MulTTiPop

Problem Statement

Automatic Music Transcription (AMT) has advanced from solo piano transcription to broader instrument and style coverage. However, a significant gap remains in multitrack transcription for real-world commercial pop music. Existing models often fail on commercially produced songs because they are trained on synthetic data (e.g., Slakh2100) or limited domains (e.g., solo piano in MAESTRO). Furthermore, systematic evaluation is hindered by a lack of datasets that pair commercial pop audio with ground truth, time-aligned multitrack transcriptions. While datasets like TheoryTab offer commercial audio, they only provide weakly aligned melody and chord annotations, not full multitrack MIDI. Conversely, datasets like RWC-Pop offer multitrack MIDI but lack genre diversity and use songs composed specifically for research rather than real-world pop.

Methodology

The authors introduce MulTTiPop, a dataset created by aligning segments of commercial pop audio with multitrack MIDI files. The construction process involves four key stages:

Metadata-Based Matching:
- Audio Source: Segments are sourced from the TheoryTab dataset (25,947 initial targets), which contains user-generated melody/chord annotations and YouTube video timestamps.
- MIDI Source: Multitrack MIDI files are sourced from the Lakh MIDI Dataset (LMD), specifically the subset matched with the Million Song Dataset (31,305 songs).
- Matching: The authors calculate the Levenshtein distance between the artist and song title metadata of TheoryTab and LMD using RapidFuzz. This filtering yields 1,164 potential matches.
Beat-Based Time Alignment:
- Since MIDI tempos often differ slightly from commercial audio, a fine-grained alignment is necessary.
- Beat Tracking: The authors use RNN-based beat tracking (via madmom) on the audio segment (with 0.5s padding) to extract an audio beat grid ( $b_a$ ). The MIDI beat grid ( $b_m$ ) is derived from the file's internal structure.
- Warping: To handle tempo discrepancies, the system performs linear interpolation on the MIDI note timings. It defines an anchor beat $k$ that links the first audio beat to the $k$ -th MIDI beat, warping the MIDI to match the audio's timing.
Candidate Anchor Beat Selection:
- Automatic anchor beat selection is found to be unreliable. The authors generate up to four candidate alignments using different metrics:
  - Base: Chroma cosine similarity and onset correlation (Pearson) between warped MIDI and audio.
  - Melody: Chroma cosine between TheoryTab melody annotations and individual MIDI instrument parts (weighted 75%).
  - YouTube Timing (yt): Filtering beats based on the absolute start time of the YouTube video relative to the full MIDI duration.
  - Combined (yt_melody): Applying the melody metric to beats filtered by the YouTube timing.
Human Annotation:
- Human annotators (students in CS and Music Technology) review the candidates. They are presented with the original audio, a piano roll visualization, a synthesized MIDI version, and an audio/MIDI overlay.
- Annotators select the alignment that best matches the audio. If no candidate is perfect, the sample is rejected. Annotators also distinguish between failures caused by incorrect metadata matching (wrong song) versus alignment issues (wrong beat/tempo).

Key Contributions

MulTTiPop Dataset: A collection of 572 segments (3.5 hours) of commercial pop music paired with time-aligned, multitrack MIDI transcriptions.
Diversity: The dataset covers 374 unique songs and 263 artists from the 1930s to the 2000s, spanning 101 genres (e.g., Rock, New Wave, Motown). This significantly exceeds the genre diversity of RWC-Pop.
Commercial Audio Ground Truth: Unlike synthetic datasets, MulTTiPop provides ground truth for fully produced commercial recordings, addressing the "synthetic-to-real" gap in AMT.
Evaluation Protocol: The dataset is split into development (169 segments) and test (403 segments) sets, stratified by artist to prevent data leakage. The authors explicitly recommend using MulTTiPop only for evaluation, not training, due to its size and copyright constraints.

Results

The authors evaluated two state-of-the-art AMT models, MT3 and YourMT3+, on the MulTTiPop test set:

Performance: Both models performed poorly compared to their results on other benchmarks (e.g., MusicNet, GuitarSet).
- YourMT3+ achieved the best Onset F1 of 37.87% (Harmonic-Percussive) and 25.29% (Exact).
- MT3 achieved an Onset F1 of 36.83% (Harmonic-Percussive) and 28.42% (Exact).
Qualitative Analysis:
- Models struggled with texturally dense music, often failing to produce coherent multitrack transcriptions.
- MT3 tended to latch onto one or two parts of a song, resulting in disjointed transcriptions and inconsistent instrument labeling.
- YourMT3+ showed better consistency but missed instrument parts and lacked rhythmic nuance. It also misidentified vocal lines (often labeling them as saxophone), which did not reflect well in "Exact" metrics.
Conclusion: The low scores indicate "substantial room for improvement" in current AMT systems when applied to real-world multitrack pop music.

Significance and Claims

The paper positions MulTTiPop as a necessary benchmark to address the stagnation of AMT on commercial pop music. Its primary significance lies in:

Bridging the Gap: Providing the first dataset of its kind that pairs commercial audio with full multitrack MIDI, enabling rigorous evaluation of models intended for real-world use.
Revealing Limitations: Demonstrating that current models, even those trained on synthetic multitrack data (Slakh2100), fail to generalize to commercial recordings, highlighting the need for better training data or architectures.
Representative Sampling: Offering a diverse sample of Western popular music use cases for multitrack AMT.

Limitations and Caveats:
The authors explicitly note a Western bias in the dataset, as the source material (TheoryTab/HookTheory) is primarily American and Western-centric. Consequently, they claim MulTTiPop reflects performance on the "Western popular music canon" rather than global music. Additionally, due to the dataset's small size and the nature of the labels (copyrighted commercial audio), the authors strictly advise against using MulTTiPop for training generative models.