⚡ electrical engineering

Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping

This paper introduces a cross-modal framework inspired by BLIP-2 that utilizes a Querying Transformer to extract implicit music styles from raw audio and condition a symbolic language model, enabling the generation of controllable and stylistically faithful piano arrangements from lead sheets and reference audio.

Original authors: Jingwei Zhao, Gus Xia, Ziyu Wang, Ye Wang

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingwei Zhao, Gus Xia, Ziyu Wang, Ye Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to play the piano. You could give it a sheet of music with notes and chords, which tells the robot what to play. But music isn't just about hitting the right keys; it's about how you hit them. It's the swing, the bounce, the gentle fade-out, and the sudden crash that makes a song feel "jazz," "classical," or "sad." These feelings are the "style" of the music. For a long time, computers have been great at reading the notes (the content) but terrible at understanding the vibe (the style). They often play a song perfectly on paper but sound like a robot reading a grocery list. This paper tackles that problem by asking: Can we teach a computer to listen to a human performance and copy the "feel" without needing a human to write down a thousand rules about what "swing" or "emotional" actually means?

The researchers behind this study, Jingwei Zhao and their team, built a clever system that acts like a musical translator. They didn't try to teach the computer style from scratch. Instead, they used two giant, pre-trained "brains" that already knew a lot about music: one that understands sound waves (audio) and one that understands musical notes (symbolic). They connected these two brains with a special middleman called a "Q-Former." Think of the Q-Former as a curious detective or a translator who listens to the audio brain describe a song's vibe and then whispers those instructions to the symbolic brain. The goal was to take a simple lead sheet (just the melody and chords) and a recording of a song played in a specific style, and have the computer generate a new piano arrangement that keeps the melody but adopts the style of the recording.

The team found that this "cross-modal bootstrapping" approach works surprisingly well. They trained their system in two stages. First, they taught the detective (the Q-Former) to listen to audio and figure out the hidden patterns of style—like the rhythm and the speed of the notes—without getting distracted by the actual notes being played. They used a method called "contrastive learning," which is like showing the detective pairs of songs and asking, "Do these two match?" to help it learn what belongs together. In the second stage, they let the detective guide the piano-playing brain to create new music. When they tested this, the system didn't just copy the notes; it captured the "groove" and the "dynamics" (how loud or soft the notes are) much better than previous methods.

In their experiments, the model was tested on thousands of songs. When asked to turn a recording into a piano arrangement, it scored higher than other top models in capturing the "feel" of the music, specifically in how well the rhythm and speed matched the original. Interestingly, while another model was slightly better at keeping the exact notes perfect for simple pop songs, this new model was much better at handling tricky, diverse genres like ballroom dancing or bossa nova. The researchers also showed that their "detective" could act as a search engine, finding the right musical style for a piece of audio even if the notes were in a different key, proving it really understood the style and not just the pitch.

Ultimately, the paper suggests that by using these pre-trained AI brains and a smart connecting layer, we can teach computers to understand the invisible, implicit rules of musical style. The system successfully generated piano covers that sounded more human and expressive, proving that we can separate the "what" of music from the "how" and teach machines to do both. The results indicate that this method is a significant step forward in making AI music generation feel less robotic and more like a genuine performance.

Technical Summary: Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping

Problem Statement
Automatic music generation has advanced significantly in handling explicit content (melody, chords, text labels), yet it struggles with implicit music style. While text descriptors like "swing" or "emotional" exist, true musical style is often an implicit quality embedded in concrete audio examples, encompassing nuanced grooving patterns, dynamics, and performance feel that abstract labels cannot fully capture. Existing audio-to-MIDI or piano cover generation models can extract pitch and timing but often fail to disentangle and reproduce the stylistic "feel" of a source performance. Current approaches typically rely on unimodal settings or fail to effectively bridge the gap between raw audio style and symbolic representation without retraining large foundational models.

Methodology
The authors propose a cross-modal framework that learns implicit music style from raw audio and applies it to symbolic piano arrangement. The core architecture utilizes a Querying Transformer (Q-Former), originally designed for vision-language tasks, to bridge a frozen pre-trained Audio Language Model (Audio LM) and a frozen pre-trained Symbolic Language Model (Symbolic LM).

The system operates in two distinct stages:

Stage I: Cross-Modal Representation Learning (Bootstrapping)
- Architecture: The Q-Former connects to a frozen Audio LM (MusicGen) via cross-attention and a symbolic stream via self-attention. A set of learnable query embeddings interacts with the Audio LM's hidden states to extract style representations ( $Z$ ).
- Training Objectives: Three complementary objectives are employed to align the modalities while preventing content leakage:
  - Audio-Symbolic Contrastive Learning: Maximizes similarity between matched audio-symbolic pairs and minimizes it for negative pairs.
  - Audio-Symbolic Matching: A binary classification task to determine if an audio-symbolic pair corresponds.
  - Audio-Grounded Symbolic Generation: An autoregressive task where the symbolic stream reconstructs a piano arrangement conditioned on the lead sheet and the style cues extracted from audio.
- Data Strategy: The model is trained on loosely aligned 10-second audio clips and 4-bar MIDI segments. To prevent memorization of exact note-to-note correspondences, the MIDI is randomly transposed and temporally shifted relative to the audio.
Stage II: Generative Modeling
- Architecture: The trained Q-Former acts as a style encoder. Its output embeddings ( $Z$ ) are concatenated with a lead sheet (providing melody and chord content) and fed into a frozen Symbolic LM (MuseCoco).
- Adaptation: Since MuseCoco does not natively support lead sheet conditioning, Low-Rank Adaptation (LoRA) adapters are inserted into its self-attention layers to incorporate the new inputs without fine-tuning the backbone.
- Output: The system generates a piano MIDI arrangement conditioned on both the content (lead sheet) and the style (extracted from the reference audio).

Key Contributions

Cross-Modal Style Alignment: The paper extends the Q-Former's role beyond vision-language content alignment to align audio and symbolic modalities via implicit music style.
Disentanglement Methodology: It presents a scalable approach to disentangle style from content using pre-trained LMs, avoiding the need for traditional latent-variable disentanglement or retraining large model backbones.
Controllable Arrangement: The model achieves style-aware audio-to-symbolic piano cover generation, outperforming existing baselines in capturing stylistic coherence while maintaining content fidelity.

Experimental Results
The model was evaluated on piano cover generation (aligned inputs) and cross-modal style transfer (unpaired inputs) using datasets like POP909, PIAST, Ballroom, and GTZAN.

Objective Metrics:
- Style Coherence: The proposed model significantly outperformed baselines (PiCoGen2, Audio-to-MIDI) in Grooving Pattern Coherence (GPC), Velocity Contour Coherence (VCC), and Tempo Accuracy (TA).
- Content Preservation: On in-distribution data (POP909), the model showed slightly lower Melody/Chord Accuracy (MCA/CA) than PiCoGen2, attributed to reliance on Sheetsage for lead sheet extraction. However, on out-of-distribution datasets (Ballroom/GTZAN), it surpassed PiCoGen2 in content preservation, demonstrating better generalization across diverse genres and instrumentation.
Subjective Evaluation: In a double-blind listening survey, the model received significantly higher ratings for Audio-to-Symbolic Coherence and Overall Musicality compared to baselines, while performing comparably in Naturalness.
Retrieval Capability: As an audio-to-symbolic retriever, the Q-Former achieved a Top-1 accuracy of 71.4% (vs. 3.4% for CLaMP3) in matching audio clips to symbolic segments, demonstrating robustness to key transposition.
Ablation Studies: Experiments confirmed that the two-stage training strategy is essential; removing Stage-I pre-training resulted in significant drops in both content preservation and style coherence. Furthermore, all three training objectives (Contrastive, Matching, Generative) were found necessary for robust alignment, particularly on complex multi-instrumental data.

Significance and Claims
The authors claim that this work demonstrates the feasibility of "bootstrapping" style transfer between unimodal large language models without retraining their backbones. By treating the Q-Former as a bottleneck for style information, the framework enables controllable, style-aware music generation that captures the "how" of a performance, not just the "what." The paper positions this as a step toward more intuitive, fine-grained control over music generation that goes beyond explicit text labels, effectively internalizing implicit stylistic traits from audio examples to drive symbolic synthesis.

More like this