⚡ electrical engineering

Repurposing Image Diffusion Models for Training-Free Music Style Transfer on Mel-spectrograms

The paper proposes Stylus, a training-free framework that repurposes pretrained image diffusion models to achieve high-fidelity music style transfer on Mel-spectrograms by manipulating self-attention mechanisms and employing phase-preserving reconstruction, thereby outperforming existing zero-shot methods in both content preservation and perceptual quality.

Original authors: Heehwan Wang, Joonwoo Kwon, Sooyoung Kim, Jungwoo Seo, Shinjae Yoo, Yuewei Lin, Jiook Cha

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heehwan Wang, Joonwoo Kwon, Sooyoung Kim, Jungwoo Seo, Shinjae Yoo, Yuewei Lin, Jiook Cha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a song you love (the Source) and you want to hear it sung in the voice of a different artist or instrument (the Style). Usually, to do this, you need a very expensive, custom-built robot that has spent years learning exactly how to mix these two things.

This paper introduces Stylus, a clever new tool that does this job without needing any extra training. It's like taking a master painter who only knows how to paint landscapes and teaching them to paint music by simply showing them the right pictures.

Here is how Stylus works, broken down into simple concepts:

1. The Big Idea: Turning Sound into Pictures

Most music AI treats sound like a long, wiggly line (a waveform). But Stylus looks at sound differently. It turns the music into a Mel-spectrogram, which is basically a colorful map or a "heat map" of the sound.

The Analogy: Think of a song as a piece of fabric. The structure (the melody and rhythm) is the pattern of the weave. The style (the instrument sound or voice) is the color and texture of the thread.
The Trick: Instead of trying to weave new fabric from scratch, Stylus treats this "sound map" like a regular image. It uses a pre-trained image AI (Stable Diffusion) that is already an expert at understanding how pixels and textures work.

2. The Magic Mechanism: Swapping the "Recipe"

The AI looks at the "sound map" and uses a part of its brain called Self-Attention. You can think of this as the AI asking itself, "What goes with what?"

The Source Song: The AI looks at the questions (Queries) the source song asks. These questions define the structure (e.g., "Where is the drum beat?").
The Style Song: The AI looks at the answers (Keys and Values) from the style song. These answers define the texture (e.g., "What does a violin sound like?").
The Swap: Stylus keeps the source song's questions but swaps in the style song's answers.
The Result: The AI draws the new picture using the structure of your original song but paints it with the texture of the new style. It's like taking a black-and-white sketch of a house and instantly coloring it to look like a watercolor painting, without changing the shape of the house.

3. Solving the "Crackling" Problem

When you change the texture of a sound map, turning it back into actual audio usually creates a lot of static, metallic crackling (like a bad radio). This happens because the AI has to guess the "phase" (the timing of the sound waves), and guessing is hard.

The Fix: Stylus is smart enough to say, "I don't need to guess the timing." It simply reuses the original timing from your source song.
The Analogy: Imagine you are remodeling a house. You change the wallpaper and the paint (the style), but you keep the original floorboards and the exact layout of the rooms (the phase). This ensures the house doesn't collapse or sound weird when you walk through it. This step is crucial for making the music sound clear and natural.

4. The "Volume Knob" for Style

Sometimes you want the new style to be subtle; other times, you want it to be overwhelming.

The Control: Stylus uses a "guidance scale" (a knob) to blend the two.
- Turn it down: The song sounds mostly like the original, with just a hint of the new style.
- Turn it up: The song takes on the new style strongly, while still keeping the original melody.
This allows for smooth mixing, like blending two colors of paint rather than just snapping between one color and another.

5. What Did They Find?

The researchers tested Stylus against other top methods using thousands of human ratings.

The Winner: Stylus was the clear champion. It kept the original song's structure much better than the others (34% better) and sounded more natural to human ears (25% better).
The Best Part: It did all this without needing to be retrained on new data. It just took an image AI, showed it some sound maps, and let it work.

In a nutshell: Stylus is a "zero-shot" magic wand. It takes an image AI, treats music maps like pictures, swaps the texture while keeping the shape, and reuses the original timing to create high-quality, style-transferred music instantly.

Technical Summary: Stylus – Training-Free Music Style Transfer via Repurposed Image Diffusion Models

Problem Statement
Music style transfer aims to blend the structural elements of a source track (melody, rhythm) with the stylistic attributes of a reference track (timbre, texture). While recent generative models have advanced music creation, existing zero-shot approaches face significant limitations. Text-based conditioning (e.g., MusicLM, Riffusion) often fails to capture the fine-grained nuances of specific audio exemplars, resulting in coarse or loosely aligned styles. Conversely, audio-to-audio diffusion methods (e.g., MusicTI) that condition directly on audio signals typically require task-specific training, fine-tuning of style encoders, or adaptation to predefined style categories. This reliance on additional training hinders scalability and prevents zero-shot deployment on novel or unlabeled timbral characteristics. Furthermore, many existing methods rely on iterative phase estimation techniques like Griffin–Lim for waveform reconstruction, which frequently introduce audible artifacts such as temporal smearing and metallic sounds.

Methodology: The Stylus Framework
The authors propose Stylus, a training-free framework that repurposes a pretrained image diffusion model (specifically Stable Diffusion v1.5) to perform music style transfer directly in the Mel-spectrogram domain. The core premise is to treat Mel-spectrograms as structured time-frequency images, leveraging the generic spatial priors learned by image diffusion models rather than the domain-specific harmonic regularities of audio-native models.

The methodology consists of three primary components:

Attention-Based Style Manipulation:
Stylus operates by manipulating the self-attention layers of the pretrained image diffusion model. It leverages the inductive bias that self-attention queries ( $Q$ ) anchor structural geometry, while keys ( $K$ ) and values ( $V$ ) encode stylistic texture.
- Process: The framework performs DDIM inversion on both the content and style Mel-spectrograms to obtain their latent representations. During the reverse generation process, the key and value features ( $K, V$ ) from the style spectrogram are injected into the self-attention layers of the content generation.
- Query Preservation: To prevent the destruction of the source musical structure, the original content queries ( $Q_{mc}$ ) are preserved and blended with the generated queries ( $Q_{mo}$ ) using a hyperparameter $\gamma$ : $\bar{Q} = \gamma Q_{mc} + (1-\gamma)Q_{mo}$ . This ensures the structural layout remains intact while the texture is modified.
- Initialization: The stylized latent noise is initialized by combining the style and content latent representations via AdaIN.
Phase-Preserving Reconstruction:
To mitigate the artifacts common in spectrogram inversion (e.g., those caused by Griffin–Lim), Stylus employs a phase-preserving strategy. Instead of estimating a new phase from the modified magnitude spectrogram, the framework reuses the phase from the original content audio. This approach ensures acoustic stability and high perceptual fidelity without requiring additional training, effectively transferring stylistic attributes through the magnitude domain while locking the phase.
Controllable Stylistic Intensity (Guidance Scaling):
To address the rigidity of simple key-value swapping, which forces a binary choice between content and style, Stylus introduces a Classifier-Free Guidance (CFG)-inspired interpolation strategy. The attention outputs for content ( $\phi_{content}$ ) and style ( $\phi_{style}$ ) are computed independently and blended using a guidance scale $\alpha$ :
$\phi_{blended} = \phi_{content} + \alpha \cdot (\phi_{style} - \phi_{content})$
This allows for continuous adjustment of stylization strength, where lower $\alpha$ values preserve structural fidelity and higher values emphasize stylistic characteristics.

Key Contributions

Training-Free Framework: The proposal of Stylus, a method that formulates music style transfer as an attention-based manipulation of Mel-spectrogram images using a fixed, pretrained image diffusion model, eliminating the need for task-specific training or fine-tuning.
Controllable Manipulation and Reconstruction: The introduction of a controllable attention manipulation mechanism (via $\alpha$ and $\gamma$ ) and a phase-preserving reconstruction strategy that jointly improve flexibility and perceptual fidelity.
Extensive Zero-Shot Evaluation: A comprehensive evaluation demonstrating that generic image priors can be effectively repurposed for the zero-shot transformation of structured Mel-spectrograms.

Experimental Results
The authors evaluated Stylus against state-of-the-art baselines, including MusicGen and MusicTI, using the MusicTI dataset (253 audio clips).

Quantitative Performance: In a large-scale evaluation of 13,246 content-style combinations, Stylus achieved a 34.1% higher score in content preservation compared to MusicTI. It also demonstrated superior performance in sound effects style fit (11.9% higher than MusicTI).
Human Evaluation: A study involving 65 valid participants and 2,925 ratings showed that Stylus significantly outperformed baselines in overall perceptual quality (3.38 vs. ≤2.69) and content preservation (4.29 vs. ≤2.95).
Ablation Studies: Experiments confirmed that removing key or value injections severely degrades performance. The ablation on the guidance scale ( $\alpha$ ) validated the trade-off between style adherence and content preservation. Furthermore, the phase-preserving reconstruction was shown to yield higher perceptual quality and musical coherence compared to Griffin–Lim reconstruction, despite a potential trade-off in fine-grained phase-dependent cues.

Significance and Claims
The paper claims that Stylus validates the hypothesis that generic spatial priors learned by image diffusion models can be robustly repurposed for the training-free transformation of structured Mel-spectrograms. By treating audio as images, the framework bypasses the limitations of audio-native models that are constrained by learned musical rules, enabling flexible, exemplar-based style transfer. The work bridges audio signal processing with generative computer vision, offering a scalable solution for personalized music creation that preserves structural integrity while achieving high-fidelity style transfer without the computational cost of additional training. The authors note that while the approach sacrifices sensitivity to fine-grained acoustic cues (such as phase dependencies) in exchange for robustness, it provides a strong foundation for future work aimed at enhancing acoustic nuance recovery while retaining training-free inference.