💻 computer science

LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training

LeVo 2 is a hybrid LLM-Diffusion framework that achieves stable, melodious, and controllable full-length song generation by employing a hierarchical modeling approach and a progressive, aesthetics-guided post-training schedule to resolve the trade-off between global semantic planning and track-specific acoustic refinement.

Original authors: Shun Lei, Huaicheng Zhang, Dapeng Wu, Yaoxun Xu, Lishi Zuo, Wei Tan, Hangting Chen, Guangzheng Li, Jianwei Yu, Zhiyong Wu, Dong Yu

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shun Lei, Huaicheng Zhang, Dapeng Wu, Yaoxun Xu, Lishi Zuo, Wei Tan, Hangting Chen, Guangzheng Li, Jianwei Yu, Zhiyong Wu, Dong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Write a Hit Song

Imagine you want to teach a robot to write a full-length song from scratch. The robot needs to do three difficult things at once:

Plan the song: Decide the melody, rhythm, and how the vocals and instruments fit together.
Write the details: Make sure the singer sounds human and the guitar sounds crisp, not robotic.
Follow instructions: If you ask for a "sad folk song about rain," it must actually sound like that, not a happy rock song.

Previous AI models struggled because they tried to do all of this in one giant, messy step. They either got the plan right but the sound quality was poor, or the sound was great but the song made no sense.

LeVo 2 is a new system that solves this by acting like a master conductor and a team of specialized musicians working together.

1. The Architecture: The Conductor and the Musicians

Instead of one brain trying to do everything, LeVo 2 splits the job into two layers, like a General and a Special Forces Unit.

The General (Mixed Semantic LM): This part looks at the big picture. It doesn't worry about the tiny details of the guitar strings yet. Instead, it plans the "skeleton" of the song: the melody, the tempo, and where the chorus goes. It creates a "mixed" plan that ensures the vocals and instruments will get along well together.
The Special Forces (Track-Specific LM): Once the General has the plan, this team takes over. They look at the plan and fill in the high-definition details. One group focuses purely on making the vocals sound perfect, while another group focuses on making the instruments sound perfect. They work in parallel, so they don't get in each other's way.
The Sound Engineer (Music Codec): Finally, a third component takes the notes from the General and the Special Forces and turns them into actual, high-quality audio waves that you can listen to.

The Analogy: Think of building a house. The General draws the blueprints (where the walls go). The Special Forces are the painters and electricians who add the fine details (the color of the paint, the wiring). The Sound Engineer is the construction crew that actually builds the house so you can live in it.

2. The Training: A Three-Step School for Musicians

The authors realized that you can't just throw a robot into a music studio and expect it to be good immediately. They created a three-stage training school guided by an "Automated Music Judge."

Stage 1: The Music Theory Class (Pre-training)
The robot listens to thousands of songs. But not just any songs—it uses an automated system to grade them. It learns from the "Top 5%" of songs first. It learns the rules of music (melody, rhythm) and gets a sense of what "good" sounds like. This is like teaching a student by showing them only the greatest hits of all time.
Stage 2: The Discipline Drill (Progressive Post-Training)
Now the robot knows music theory, but it might still make mistakes, like singing the wrong words or ignoring your instructions.
- First, it practices Supervised Fine-Tuning (SFT): It only practices on the absolute best songs to get the quality high.
- Next, it does Offline DPO: This is like a strict coach. The robot generates many versions of a song, and the coach picks the one that follows the lyrics best and the one that sounds most like the prompt. The robot learns to stop "hallucinating" (making up fake words) and start listening.
- Finally, it does Semi-Online DPO: The robot starts generating its own new songs and learns from its own improvements, pushing its artistic creativity higher without losing its discipline.
Stage 3: The Masterclass (Modular Extension)
The General (the planner) is now perfect and is frozen (not changed). The Special Forces (the detail team) get extra training. They practice taking a rough sketch and turning it into a crystal-clear, high-fidelity recording. This ensures the vocals and instruments sound rich and detailed.

3. The "Aesthetic Guide"

A key innovation in this paper is the Automated Music Aesthetic Evaluation Framework. Imagine a robot judge that listens to a song and gives it a score for "Musicality."

During training, this judge tags songs with "Musicality Tiers" (e.g., "Top Tier," "Average," "Low").
The robot learns that when it sees the "Top Tier" tag, it should try to generate something amazing.
This helps the robot understand why some songs are better than others, rather than just copying them blindly.

4. The Results: How Good Is It?

The authors tested LeVo 2 against other open-source models and even some famous commercial systems (like Suno and Mureka).

Better than open-source: LeVo 2 beat all other open-source models in almost every category, including melody, arrangement, and sound quality.
Rivaling the pros: It came very close to the performance of the top commercial systems, which usually keep their secrets hidden.
Following instructions: It was very good at singing the correct lyrics and matching the mood (emotion) you asked for, significantly reducing the "hallucinations" where AI makes up nonsense words.

Summary

LeVo 2 is a new way to make AI music. Instead of one brain trying to do everything at once, it uses a hierarchical team (a planner and detail specialists) and a step-by-step training school that teaches the AI to first understand music theory, then learn to follow rules, and finally polish the sound to perfection. The result is a system that can generate full-length, coherent, and high-quality songs that sound surprisingly human.

Technical Summary: LeVo 2

Problem Statement

Full-length song generation presents a fundamental structural trade-off in existing language model-based systems. Approaches utilizing mixed-token modeling (predicting vocals and accompaniment simultaneously) preserve vocal-instrument coordination but obscure track-specific acoustic details due to mutual masking and restricted discrete vocabularies. Conversely, dual-track prediction strategies improve acoustic fidelity but drastically increase sequence length, weakening global planning capabilities and making long-term structural coherence difficult to maintain.

Furthermore, the field is constrained by a scarcity of high-quality data and noisy annotations. Existing preference alignment methods often suffer from gradient conflicts when simultaneously optimizing for multiple objectives (e.g., controllability vs. musicality), leading to an averaging effect that limits the upper bound of individual capabilities. Additionally, reliance on static, offline preference datasets restricts model performance to the capabilities of the data generators and renders training susceptible to reward hacking.

Methodology

LeVo 2 addresses these challenges through a hybrid LLM-Diffusion framework that employs hierarchical representation modeling and a decoupled, aesthetics-guided training paradigm.

1. Hierarchical Architecture (LeLM)

The core of the system is the LeVo Language Model (LeLM), which separates global semantic planning from track-specific acoustic refinement:

Mixed Semantic LM: A decoder-only Transformer that predicts mixed tokens. These tokens serve as a "pseudo-chain-of-thought," capturing high-level structural information (melody, rhythm, tempo) and ensuring global vocal-instrument harmony without expanding sequence length.
Track-Specific LM: A lightweight module that operates in parallel, conditioned on the hidden states of the Mixed Semantic LM. It predicts vocal and accompaniment tokens separately to recover fine-grained acoustic details. To provide comprehensive context, a delay pattern mechanism is employed, allowing the model to access future hidden states from the semantic planner during dual-track prediction.
Music Codec: A diffusion-based decoder reconstructs the predicted dual-track tokens into full-length, high-fidelity waveforms. It utilizes a pre-trained encoder (MuEncoder) and specialized Vector Quantizers (VQs) to discretize audio into tokens.

2. Aesthetics-Guided Three-Stage Training Paradigm

LeVo 2 introduces a novel training schedule that injects musicality priors and decouples optimization objectives:

Stage 1: Aesthetics-Conditioned Pre-training:
- The Mixed Semantic LM is trained on large-scale data using an automated music aesthetic evaluation framework. This framework assigns "musicality-tier" conditions (e.g., Top 5%, 5-25%) to training data based on predicted quality scores.
- This allows the model to learn core acoustic and semantic knowledge from noisy, long-tail data while establishing a prior for musicality, enabling musicality-aware Classifier-Free Guidance (CFG) during inference.
- Pure instrumental music is included to strengthen accompaniment modeling.
Stage 2: Decoupled Progressive Post-training:
- Supervised Fine-Tuning (SFT): Filters data to the top 0.5% of aesthetic scores to establish a high-quality generation baseline and narrow the output distribution.
- Large-Scale Offline DPO: Focuses on controllability. Using 150,000 diverse lyrics and prompts, the model is aligned to reduce lyrical hallucinations and improve prompt consistency (genre, emotion, instruments).
- Closed-Loop Semi-Online DPO: Focuses on musicality. A periodically updated generator samples new data to create preference pairs, pushing the model beyond the limits of static offline datasets while preventing reward hacking through hybrid validation.
Stage 3: Aesthetics-Conditioned Modular Extension:
- The weights of the aligned Mixed Semantic LM are frozen. The Track-Specific LM is trained exclusively to refine acoustic details.
- An acoustic augmentation strategy is employed: mixed tokens are reconstructed into low-quality audio (via limited diffusion steps) and re-tokenized. This forces the Track-Specific LM to learn how to recover and supplement fine-grained acoustic nuances.

Key Contributions

Hierarchical Representation Modeling: A novel architecture that decouples global semantic planning (Mixed Semantic LM) from track-specific acoustic refinement (Track-Specific LM), resolving the trade-off between structural coherence and acoustic fidelity without sequence length explosion.
Aesthetics-Guided Training: The introduction of an automated music aesthetic evaluation framework to assign musicality-tier conditions during pre-training, providing musicality priors before preference alignment.
Decoupled Progressive Post-Training: A three-stage alignment strategy (SFT $\to$ Offline DPO for controllability $\to$ Semi-Online DPO for musicality) that mitigates gradient conflicts and avoids the limitations of static offline datasets.
Modular Extension with Acoustic Augmentation: A specialized training phase for the Track-Specific LM that uses intentional acoustic corruption to enhance the model's ability to restore fine-grained details.

Results

Extensive evaluations demonstrate that LeVo 2 significantly outperforms existing open-source baselines and approaches leading commercial systems:

Subjective Evaluation: In a six-dimensional Mean Opinion Score (MOS) test involving music professionals, LeVo 2 surpassed all open-source models (YuE, DiffRhythm 2, ACE-Step, HeartMuLa, LeVo) across Overall Musicality, Melody, Arrangement, Structure, Vocal Sound Quality, and Accompaniment Sound Quality.
Commercial Comparison: LeVo 2 approaches the performance of proprietary systems like Suno v5 and Mureka v8 in metrics such as Melody (6.12 vs. 6.38), Arrangement (6.42 vs. 6.55), and Accompaniment Sound Quality (7.10 vs. 7.11). Notably, it surpasses MiniMax Music 2.5+ across all subjective listening metrics.
Objective Evaluation: LeVo 2 achieved the lowest Phoneme Error Rate (PER) among open-source systems (8.55%), indicating superior lyric alignment. It also achieved the highest Emotion control score (8.72) among all evaluated systems, including commercial ones.
Ablation Studies: Experiments confirmed that removing the hierarchical architecture, the aesthetic guidance, or the progressive post-training stages leads to significant performance degradation, validating the necessity of each component.

Significance

The paper claims that LeVo 2 represents a significant advancement in automated song generation by successfully addressing the structural trade-offs inherent in previous models. By separating the optimization of controllability and musicality into distinct stages and utilizing a hierarchical architecture, the framework mitigates gradient conflicts and maximizes the performance upper bound of the model. The authors posit that LeVo 2 demonstrates that open-source architectures can approach commercial standards in full-length song generation, particularly when guided by aesthetic priors and decoupled training strategies. The release of the model weights and code is intended to facilitate further research in high-fidelity, controllable music generation.