🤖 machine learning

MuScriptor: An Open Model for Multi-Instrument Music Transcription

MuScriptor is an open-weight model that achieves effective multi-instrument music transcription on real-world recordings by combining synthetic pre-training, real-data fine-tuning, reinforcement learning, and instrument-conditioning to overcome the generalization limitations of existing methods.

Original authors: Simon Rouard, Michael Krause, Axel Roebel, Carl-Johann Simon-Gabriel, Alexandre Défossez

Published 2026-07-10✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simon Rouard, Michael Krause, Axel Roebel, Carl-Johann Simon-Gabriel, Alexandre Défossez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot to listen to a chaotic rock concert and write down exactly which musician played which note, on which instrument, and for how long. That's the job of Automatic Music Transcription (AMT). For years, scientists have built robots that are great at transcribing a solo piano or a single drum kit, but as soon as you throw in a full band with electric guitars, distorted bass, and a screaming singer, the robots usually crash and burn. They get confused by the noise, the overlapping sounds, and the messy reality of real-world music.

Enter MuScriptor, a new open-source model that claims to be the first to successfully tackle this "full band" chaos across almost every genre, from classical orchestras to heavy metal.

The Secret Sauce: Fake Practice vs. Real Gigs

The paper's main discovery is a specific recipe for training these music-reading robots. The authors found that you can't just feed a robot real music and expect it to learn quickly, but you also can't just feed it perfect, computer-generated music and expect it to work in the real world.

Think of it like training a chef.

Synthetic Data (The Simulation): The team started by feeding the model 1.45 million computer-generated MIDI files. This is like a chef practicing in a perfect, sterile kitchen where every ingredient is measured exactly, and nothing ever burns. The paper suggests this is excellent for "pre-training"—giving the robot a huge head start on the basics of music theory.
Real Data (The Restaurant): However, the paper explicitly argues that stopping at the simulation is a failure. If you only train on fake data, the robot fails miserably when it hears a real guitar with distortion or a singer with reverb. To fix this, the team added 170,000 real music recordings (totaling over 11,000 hours) to the mix. This is the "real restaurant" where the chef learns to handle messy, unpredictable ingredients.
The Result: The paper shows that combining the two is the magic. The model trained only on fake data was okay at recognizing notes but terrible at the details. But once they fine-tuned it on the real recordings, the performance jumped by roughly 20 percentage points across the board.

The "Reinforcement Learning" Polish

After the robot learned from the fake and real data, the authors gave it one final, very strict polish. They took a tiny, super-high-quality set of 300 songs where the notes were perfectly annotated by humans. They used a technique called Reinforcement Learning (specifically a method called GRPO) to let the model practice on these 300 tracks and "learn from its mistakes" by comparing its output to the perfect human version.

Think of this as the chef tasting their own dish and realizing, "Oh, I added too much salt," and adjusting. The paper suggests this step didn't just tweak the numbers; it specifically reduced the number of notes the model missed (false negatives) and made the timing of the notes sharper.

What the Robot Can (and Can't) Do

The paper is very clear about what MuScriptor can do. It can listen to a 5-second clip of audio and spit out a list of notes. It can even be told, "Hey, I know there's a guitar and a drum in this song, but ignore everything else." This "instrument conditioning" helps the robot stay focused and not get confused about which instrument is playing which note.

However, the paper also rules out a few things:

It's not magic for overlapping notes: If two people play the exact same note on the same instrument at the exact same time (like two violins playing a unison), the current system has to drop the shorter one to count it correctly. The paper admits that a different "tokenization" (a different way of writing down the notes) would be needed to handle this perfectly.
It's not a "one-size-fits-all" for every single dataset yet: While it beats the previous best models (like YourMT3+) on many tests, the paper notes that on some specific datasets (like choral music), the timing of when a note starts and stops is still tricky. The paper suggests that annotating precise start and end times for certain styles of music is inherently difficult, even for humans.

The Numbers Game

The paper doesn't just say "it's better"; it measures it.

On a test set of diverse songs, the final model (trained on synthetic data, real data, and the 300-song polish) achieved a Multi F1 score of 47.8 (with a specific setting called αCFG = 2).
Compare that to the previous best model, YourMT3+, which scored 21.9 on the same metric. That's a massive leap.
Even the smallest version of their model (with 60 million parameters) managed a frame F1 score of 65, suggesting that you don't need a supercomputer to get decent results.

The Bottom Line

The paper concludes that MuScriptor is a significant step forward because it's an open-weight model (meaning anyone can download and use it) that actually works on real, messy, multi-instrument music. It suggests that while synthetic data is a powerful tool for the early stages of learning, it is not sufficient on its own. You need the messy, real-world data to teach the robot how to handle the chaos of a real concert.

The authors are confident in their results because they tested them on a diverse set of genres and compared them directly against state-of-the-art baselines. They don't claim to have solved every problem in music transcription (like perfectly handling overlapping notes on the same instrument), but they have built a tool that is far more useful for musicians and researchers than anything that came before it.

Technical Summary: MuScriptor – An Open Model for Multi-Instrument Music Transcription

1. Problem Statement

Automatic Music Transcription (AMT) aims to convert audio recordings into symbolic representations (typically MIDI). While significant progress has been made for single-instrument scenarios (e.g., piano, guitar, drums), general-purpose transcription for multi-instrument, real-world music mixes remains a critical bottleneck. Existing state-of-the-art models, such as MT3 and its derivatives, often rely heavily on synthetic training data. While these models perform well on synthetic test sets, they suffer from a significant domain shift when applied to complex, real-world productions featuring diverse timbres, overlapping frequencies, and audio effects (e.g., distortion). Consequently, current multi-instrument AMT systems are often too error-prone for downstream applications in musicology, generative modeling, or music information retrieval (MIR).

2. Methodology

2.1 Model Architecture

MuScriptor employs a decoder-only Transformer architecture, avoiding complex encoder-decoder or hierarchical attention structures in favor of a simple yet effective design.

Inputs: The model accepts a 5-second mel-spectrogram (derived from 16kHz mono audio via STFT with 2048 nFFT, 160 hop size, and 512 mel bins) and an optional list of target instruments.
Conditioning: Instrument presence is provided as a global conditioning signal (the full track's instrument list) rather than local segment data. These instrument embeddings are concatenated with the spectrogram projection to serve as prefix conditioning for the Transformer.
Tokenization: Note annotations are tokenized similarly to MT3, mapping 128 MIDI instruments into 36 subgroups based on the MT3_FULL_PLUS taxonomy.
Inference: Transcriptions are generated via autoregressive decoding. The system supports Classifier-Free Guidance (CFG) with a strength of $\alpha_{CFG} = 2$ to improve generation quality.

2.2 Data Strategy

The training pipeline utilizes a three-stage approach combining synthetic and real data:

Synthetic Pre-training ( $D_{Synth}$ ): A dataset of 1.45 million MIDI files is converted to audio on-the-fly. The pipeline applies random augmentations (pitch shifting, tempo changes, velocity adjustments, instrument randomization) and synthesizes audio using a pool of 250+ soundfonts with random detuning. This creates an infinite variety of audio realizations from the symbolic data.
Real Data Fine-tuning ( $D_{Real}$ ): The model is fine-tuned on a curated dataset of 170,000 real music recordings (approx. 11,000 hours) covering diverse genres from classical to heavy metal. These tracks feature aligned note annotations obtained via audio-symbolic synchronization (using linear interpolation and dynamic time warping).
Reinforcement Learning Post-training ( $D_{RL}$ ): A small subset of 300 high-quality, manually verified tracks is used for post-training. The model employs a Group Relative Policy Optimization (GRPO) algorithm.
- For each batch, the model generates $G=8$ independent transcriptions.
- Rewards are calculated as the sum of Onset, Frame, and Offset F1 scores against the ground truth.
- Advantages are computed by normalizing rewards within the group, and the model is updated via a REINFORCE objective weighted by these advantages.

2.3 Evaluation Metrics

Performance is measured using standard MIR metrics from the mir_eval library:

Onset F1: Pitch match with $\pm 50$ ms onset tolerance.
Offset F1: Includes onset criteria plus offset tolerance of $\max(50\text{ms}, 0.2 \times d_{ref})$ .
Frame F1: Frame-wise overlap at 62.5 ms resolution.
Multi F1: The most stringent metric, requiring correct pitch, timing, and instrument classification.
Note: Overlapping notes of the same instrument and pitch are removed from the test set prior to evaluation due to tokenization limitations.

3. Key Contributions

MuScriptor Model: The release of an open-weight, general-purpose multi-instrument transcription model (available in sizes from 60M to 1.3B parameters) capable of handling diverse musical genres.
Large-Scale Dataset: The curation of a comprehensive dataset comprising 170k real-world recordings with aligned annotations and 1.45M synthetic MIDI files, addressing the scarcity of multi-instrument annotated data.
Analysis of Synthetic Pre-training: A comparative study demonstrating that while synthetic data is insufficient for standalone general-purpose transcription, it is highly effective for pre-training, particularly when real data is scarce.
Reinforcement Learning Alignment: The application of a GRPO-like algorithm to align the model with high-quality transcriptions, a technique previously unexplored in AMT literature.
Instrument-Conditioned Transcription: The introduction of optional instrument conditioning, allowing users to customize output (e.g., transcribing specific instruments) and stabilizing predictions across segment boundaries.

4. Experimental Results

4.1 Training Stages and Performance

Experiments on the 1.3B parameter model (evaluated on a held-out test set $D_{Test}$ ) show a clear progression in performance:

Synthetic Only: Achieves competitive Frame F1 (51.3) but poor Multi F1 (15.2) and Onset F1 (26.1), confirming the domain gap.
Synthetic + Real ( $D_{Synth} + D_{Real}$ ): Fine-tuning on real data yields a massive improvement, raising Multi F1 to 41.7 and Onset F1 to 54.4.
Full Pipeline ( $D_{Synth} + D_{Real} + D_{RL}$ ): The addition of RL post-training further boosts performance, achieving a Multi F1 of 47.8 and Onset F1 of 60.4 (with $\alpha_{CFG}=2$ ).

4.2 Comparative Analysis

vs. State-of-the-Art: MuScriptor significantly outperforms the YourMT3+ baseline (a strong SOTA model) across multiple external benchmark datasets (e.g., Bach10, Dagstuhl ChoirSet, RWC). For instance, on the Dagstuhl ChoirSet, Frame F1 improves from 51.0 (YourMT3+) to 80.7 (MuScriptor).
Synthetic Pre-training Impact: Analysis shows pre-training is most beneficial when real data is limited (e.g., using only 1% of $D_{Real}$ , pre-training improves Offset F1 from 9.9 to 33.4). However, as the volume of real data increases, the relative gain from pre-training diminishes, though it still provides a slight edge at 100% data usage.
Model Scaling: Performance scales with model size; the 60M parameter model achieves a Frame F1 of 65.7, while the 1.3B model reaches 68.7, suggesting applicability across different resource constraints.
Input Representations: Mel-spectrograms outperformed alternative inputs like Magnitude CQT, Encodec codecs, and MERT embeddings, suggesting that representations closer to raw signal properties are optimal for this task.

5. Significance and Claims

The paper claims that MuScriptor is the first effective open-weight model for multi-instrument music transcription across a wide variety of genres.

The authors emphasize that while previous approaches relied on architectural refinements (e.g., hierarchical attention, MoE), their success stems primarily from data scale and type—specifically the strategic combination of large-scale synthetic pre-training, extensive real-world fine-tuning, and RL-based alignment. The model is presented as a foundational tool for musicians, musicologists, and researchers in MIR and generative modeling, enabling tasks previously hindered by the lack of accurate transcription tools for complex mixes. The authors modestly note that limitations remain regarding the tokenization of overlapping notes for the same instrument and suggest future work could expand the instrument vocabulary and segment sizes.