💬 NLP

MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models

The paper introduces MIDI-LLM, a two-stage training framework that adapts Large Language Models to generate high-quality, text-controlled multitrack MIDI music and lead sheets, demonstrating superior performance and user acceptance over existing baselines through extensive ablation studies and a large-scale real-world blind evaluation.

Original authors: Shih-Lun Wu, Dave Carlton, Ryan Miyakawa, Yoon Kim, Chris Donahue, Cheng-Zhi Anna Huang

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Shih-Lun Wu, Dave Carlton, Ryan Miyakawa, Yoon Kim, Chris Donahue, Cheng-Zhi Anna Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can talk to a computer and ask it to write a song for you. In the realm of artificial intelligence, this is called "text-to-music." For a while, the best way to do this was to ask the computer to generate raw sound waves, like a digital recording. But here's the catch: once that sound is made, it's like a painting on a canvas. You can't easily erase a single brushstroke or change the tempo of just the drumbeat without ruining the whole picture. Musicians need something more flexible, something they can edit note-by-note. This is where "MIDI" comes in. Think of MIDI not as sound, but as a digital sheet music or a set of instructions telling a virtual piano exactly which keys to press, how hard, and for how long. It's the difference between a finished recording and a Lego set; you can take the Lego apart and build something new.

Now, imagine taking the super-smart language models (the AI that writes essays and answers questions) and teaching them to speak this musical language. That's the big idea behind this new research. The scientists wanted to see if they could turn a text expert into a music expert, allowing people to type things like "a sad jazz song with a fast drum" and get a perfectly editable musical score in return. They weren't just trying to make a pretty noise; they wanted to build a tool that helps human creators brainstorm and collaborate with AI in a way that feels natural and controllable.

The Paper: MIDI-LLM

The researchers behind this paper, a team from MIT, Hooktheory, and Carnegie Mellon University, have cooked up a new recipe called MIDI-LLM. Think of it as a way to upgrade a standard Large Language Model (LLM)—the kind of AI that writes stories and solves math problems—so it can also write music.

Usually, these AI models are like chefs who only know how to cook with words. They understand "apple" and "pie," but they don't know what a "C-sharp note" sounds like or how to write it down. The team's first move was to give the AI a new vocabulary. They expanded the model's dictionary to include special "MIDI tokens." Instead of writing out a note as a long sentence like "start at 10 seconds, last for 0.5 seconds, play a high C," the model now sees it as a single, compact symbol, like a secret code. This is like teaching a chef to read a new language where a single word means "add salt," making the cooking process much faster and more efficient.

But just giving the model a new dictionary isn't enough; it needs to learn how to use it. The team trained the AI in two distinct stages, like a music student first learning theory and then playing in a band.

Stage 1: The Solo Practice (Unimodal Pretraining)
First, they let the AI practice on its own. They fed it two types of data:

Music-related text: Articles, questions, and answers about music theory (like "What is a minor chord?").
Standalone MIDI files: Thousands of raw musical scores without any text descriptions.
This stage was about teaching the AI the "syntax" of music. It learned how notes fit together, how rhythms work, and the structure of a song, all without needing a human to tell it what to do. It's like a musician practicing scales and reading sheet music in a quiet room.

Stage 2: The Duet (Multimodal Supervised Finetuning)
Next, they paired the AI with a partner. They fed it pairs of text and music: a prompt like "a happy pop song" followed immediately by the corresponding MIDI score. This taught the AI to translate human ideas into musical instructions. If you said "jazz," it learned to write jazz notes. If you said "sad," it learned to write slow, minor chords. This is the moment the AI learns to listen to your request and actually play what you asked for.

What They Found

The results were pretty impressive. When they tested their new MIDI-LLM against a recent competitor called Text2midi, their model came out on top in two major ways:

Better Quality: The music it generated sounded more realistic and followed the rules of music better.
Better Control: It listened more closely to the text instructions. If you asked for a specific mood or genre, it actually delivered.
Speed: Because they used a standard AI architecture (based on Llama 3.2), they could use existing, super-fast computer tools to run the model. Their model was 7 to 13 times faster than the competitor, even though it was slightly larger.

They also tested a special feature called "infilling." Imagine you have a song that's half-finished, and you want the AI to write the middle part. The team found that the AI was great at this, but there was a tricky balance. If the AI was too focused on the text you typed, it might ignore the music you had already written. If it was too focused on the existing music, it might ignore your new text instructions. To fix this, they introduced a "knob" (called Classifier-Free Guidance) that lets users adjust how much the AI should listen to the text versus the existing music.

Real-World Testing: The "In-the-Wild" Study

To see if this actually helped real people, the team didn't just run computer tests; they went into the real world. They partnered with Hookpad, a website where songwriters create music. They let 58 real users (who were already subscribers) try out their new AI co-pilot.

The users were asked to create songs from scratch or fill in missing parts of songs. They could choose from three different AI models:

The old model (which only looked at the music, ignoring text).
A version of the new model that ignored text.
The full new MIDI-LLM (which listened to text).

The results showed that when users were starting a song from scratch ("zero-to-one"), the full MIDI-LLM was the clear winner. It had nearly double the acceptance rate of the other models. This suggests that when people are trying to get a new idea going, being able to type "make it sound like a 90s rock song" is incredibly valuable.

However, when users were just filling in a small gap in an existing song, the results were more mixed. Sometimes, the users preferred the older model that ignored the text. The researchers suggest this might be because, when you are already deep in a song, you might want the AI to just "fit in" with what's already there, rather than trying to force a new idea that might clash with the existing melody.

The Bottom Line

The paper concludes that adapting Large Language Models for music is a powerful recipe. By teaching these models to speak both "text" and "MIDI," they created a tool that helps humans and AI collaborate more effectively. It's not just about generating noise; it's about giving creators a flexible, fast, and intelligent partner that can turn a simple sentence into a full musical score, ready for them to tweak and perfect. The team is already rolling this out to more users, hoping to see how it changes the way people write music in the future.

Technical Summary: MIDI-LLM

Problem Statement

While recent text-to-music models have enabled the generation of realistic audio from natural language prompts, their audio waveform outputs remain difficult to precisely edit, restructure, or reuse in downstream creative workflows. This limitation hinders the iterative human-AI collaboration that Large Language Models (LLMs) have successfully enabled in text domains. Conversely, symbolic-domain models that generate MIDI outputs allow for direct editing and reuse, promoting creative agency. However, existing symbolic models often lack effective free-form text control, limiting their utility for rapid ideation and instruction-based editing. Furthermore, many symbolic models rely on custom architectures that require bespoke engineering efforts to accelerate inference for production-scale deployment, unlike the well-developed inference ecosystems available for open-weight LLMs.

Methodology

The paper proposes MIDI-LLM, a recipe for adapting LLMs to improve text-to-MIDI generation and infilling. The approach involves three core components:

1. Vocabulary Expansion

To incorporate MIDI data into an LLM without the sequence length overhead of serializing tokens as text strings, the authors expand the token embedding layer of the Llama 3.2 (1B) model. They integrate the Anticipatory Music Transformer (AMT) vocabulary, which represents each musical note as a triplet of tokens:

Onset time: Quantized in 10ms steps (0–100s).
Duration: Quantized in 10ms steps (0–10s).
Instrument-pitch joint token: A combination of 129 instruments and 128 pitches.
This results in a vocabulary expansion from the original text size to include 55K new MIDI tokens. The new embedding weights and the corresponding unembedding layer are randomly initialized.

2. Two-Stage Training Pipeline

The model undergoes a two-stage training process totaling over 4 billion tokens:

Stage I: Unimodal Continued Pretraining (CPT): The model is trained on a mix of music-adjacent text (e.g., music articles, GPT-4 synthetic knowledge, ABC notation) and standalone multitrack MIDI data (GigaMIDI). This stage aims to surface existing musical knowledge within the LLM and teach the syntax and structure of MIDI data under the AMT framework.
Stage II: Multimodal Supervised Finetuning (SFT): The model is finetuned on paired text-MIDI data (MidiCaps + Lakh MIDI). Text prompts describing genre, mood, instrumentation, and chords serve as instructions to generate corresponding MIDI sequences. This stage also includes span infilling tasks where the model predicts missing notes based on surrounding MIDI context and text instructions.

3. Downstream Specialization and Inference

The resulting model serves as a foundation for downstream tasks, specifically demonstrated on the TheoryTab dataset for lead sheet (melody + chords) generation and infilling.

Classifier-Free Guidance (CFG): To address the issue where surrounding MIDI context dominates text prompts during infilling, the authors implement CFG. This allows users to trade off quality for stronger text control by modulating a scale parameter ( $\lambda$ ) during inference.
Optimized Inference: By preserving the original LLM parameter structure, the model integrates seamlessly with optimized inference ecosystems like vLLM, utilizing techniques such as CUDA graphs, paged KV cache attention, and FP8 quantization.

Key Contributions

A Recipe for Adaptation: A methodology for adapting LLMs to text-to-MIDI generation via vocabulary expansion and a two-stage training pipeline (unimodal CPT + multimodal SFT) that integrates with existing LLM serving ecosystems.
Concrete Instantiation: A specific implementation based on Llama 3.2 (1B) that outperforms the recent Text2midi model in both text control and musical quality.
Specialization for Lead Sheets: Demonstration that the MIDI-LLM can be specialized for lead sheet composition (melody + chords) and an ablation study validating the synergy between text pretraining, MIDI pretraining, and text-to-MIDI supervised finetuning.
Real-World Validation: An "in-the-wild" user study conducted with 58 Hookpad Aria users and 4,002 generated outputs, evaluating the model's efficacy in human-AI co-creation.

Results

Quantitative Benchmarks

vs. Text2midi: On the MidiCaps + Lakh MIDI dataset, MIDI-LLM achieves higher music quality (lower FAD) and stronger text control (higher CLAP scores) compared to Text2midi.
Inference Speed: Despite being 5x larger in parameter count, MIDI-LLM achieves 7–13x faster inference than Text2midi due to vLLM integration.
Ablation Study: On the TheoryTab downstream task, the full recipe (combining Llama text pretraining, unimodal CPT, and SFT) achieves the best balance of music quality and textual controllability.
- Text-only pretraining establishes strong control but yields lower musical quality.
- MIDI-only pretraining improves quality but results in poor or inconsistent text control.
- The combination of modalities reveals a structural synergy where broad textual knowledge enhances both control and musical quality.

User Study Findings

Zero-to-One Generation: In "From Scratch" lead sheet generation, MIDI-LLM achieved a 13.0% acceptance rate, nearly double the 7–8% rate of baselines (AMT without text and MIDI-LLM without text). This indicates high user value for text control when establishing new musical ideas.
Infilling: For infilling tasks (melody, chords, or both), the MIDI-only baseline (AMT) performed competitively or better. The authors attribute this to potential context-control mismatches or user preferences for musical continuity over strict text adherence in existing contexts.
Post-Study Impact: Following the study, broad deployment of MIDI-LLM to Hookpad Aria users resulted in a 22% increase in accepted generations and infills.

Significance and Claims

The paper claims that MIDI-LLM successfully imbues symbolic models with effective free-form text control while leveraging the efficiency and ecosystem of open-weight LLMs. By demonstrating that a unified LLM architecture can outperform custom symbolic models in both quality and control, the work suggests that the "recipe" of adapting LLMs is a viable path for music generation.

The authors emphasize that their approach supports the kind of iterative human-AI collaboration seen in text domains, which audio waveform models struggle to provide. The integration of CFG allows for tunable control over the tension between pre-existing musical context and user intent, a critical feature for creative workflows. Finally, the successful deployment in a real-world co-creation platform (Hookpad) validates the practical utility of text-conditioned symbolic music generation for "zero-to-one" music ideation.