⚡ electrical engineering

Aligning Language Models for Lyric-to-Melody Generation with Rule-Based Musical Constraints

This paper proposes a novel alignment framework that automatically generates rule-based musical constraints to train Large Language Models via Direct Preference Optimization and Kahneman-Tversky Optimization, effectively reducing musical implausibilities and improving lyric-to-melody generation quality without requiring human annotation.

Original authors: Hao Meng, Siyuan Zheng, Shuran Zhou, Qiangqiang Wang, Yang Song

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Meng, Siyuan Zheng, Shuran Zhou, Qiangqiang Wang, Yang Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot that has read every song ever written. You ask it to write a new song for you, giving it just the lyrics. Because it's so smart, it can write lyrics that rhyme and make sense. But here's the problem: when it tries to turn those words into a tune, the music sounds weird. The notes might jump around like a broken record, the rhythm might be impossible to clap along to, or the singer might have to stretch their voice to an impossible high note just to hit one note.

This paper is about teaching that robot to stop writing "musical nonsense" and start writing songs that actually sound good, without hiring a team of human music teachers to correct it one by one.

Here is the story of how they did it, broken down into simple steps:

1. The Problem: The Robot is "Rule-Breaking"

The researchers started with a standard AI model (a Large Language Model) that had been trained on thousands of songs. They called this the SFT Model (Supervised Fine-Tuning).

The Analogy: Think of this robot like a student who has memorized the answers to a math test but doesn't actually understand the rules of math. It can guess the right answer often, but sometimes it writes "2 + 2 = 5" or draws a triangle with four sides.
The Issue: In music, these "mistakes" are called Constraint Violations. The robot might write a melody where the singer has to hold a note for 10 minutes straight, or where the pitch jumps from a baby's squeak to a lion's roar instantly. It's technically "music," but it's not singable or pleasant.

2. The Solution: The "Rulebook" Instead of a Human Teacher

Usually, to fix a robot, you need humans to listen to its songs and say, "This one is good, that one is bad." This is slow, expensive, and boring.

The Innovation: Instead of hiring humans, the researchers wrote a digital rulebook. They created 5 simple, strict rules that any good song must follow:
1. Format: The song must be written in the right computer code.
2. Lyrics: The words must match the notes (no singing gibberish).
3. No Boredom: Don't just repeat the exact same note 50 times in a row (that's boring).
4. Rhythm: Notes shouldn't be too short to hear or too long to breathe.
5. Range: The song must be singable by a normal human voice (no alien frequencies).

3. The Training: The "Auto-Grader" System

Now, they needed to teach the robot these rules without humans. They built an Auto-Grader.

How it worked: They asked the robot to write 20,000 songs. Then, they ran every single song through their digital rulebook.
- If a song broke a rule, the grader marked it as a "Loser."
- If a song followed the rules, it was a "Winner."
The Magic: They didn't need a human to listen. The computer knew instantly if the song was "illegal" because it broke the rules. This created a massive dataset of "Good vs. Bad" examples automatically.

4. The Two-Step Dance: DPO and KTO

To teach the robot, they used a special two-step training method. Think of this like training a dog.

Step 1: DPO (Direct Preference Optimization) - The "Compare and Choose"
- The Analogy: You show the dog two treats. One is a delicious steak (the "Winner" song), and one is a piece of cardboard (the "Loser" song). You say, "I like the steak, not the cardboard." The dog learns to prefer the steak.
- In the paper: The AI looks at pairs of songs (one good, one bad) and learns to pick the one that follows the rules.
Step 2: KTO (Kahneman-Tversky Optimization) - The "Don't Do That!"
- The Analogy: Sometimes, the dog doesn't have a "good" option to compare against; it just does something silly, like chewing a shoe. You can't show it a "good shoe," you just have to say, "No! Bad dog!"
- In the paper: Sometimes the robot fails so badly it can't even make a "good" song to compare. In these cases, the researchers just told the AI, "This output is bad, don't do it again." This step cleans up the specific mistakes the robot keeps making.

5. The Result: A Musical Genius

After this training, the robot was transformed.

Before: It wrote songs that sounded like a broken music box.
After: It wrote songs that humans rated as nearly as good as real, human-composed music.
The Proof: When they tested the new robot, it made far fewer mistakes. It didn't write impossible notes, the rhythm was natural, and the lyrics matched the tune perfectly.

The Big Takeaway

This paper proves that you don't need a million human music critics to teach an AI to write good songs. You just need to give the AI a clear set of rules (like a rulebook for a game) and let the computer grade itself.

It's like teaching a child to drive: instead of a human sitting in the passenger seat screaming "Stop!" every time they make a mistake, you put up clear signs (Stop signs, speed limits) and let the car's computer learn to follow them. The result is a driver (or in this case, a songwriter) that is safe, reliable, and ready for the road.

1. Problem Statement

While Large Language Models (LLMs) have shown promise in generating melodies from lyrics, models trained via standard Supervised Fine-Tuning (SFT) often suffer from "constraint violations." These models learn statistical patterns from training data but lack robust adherence to fundamental musical principles. Consequently, SFT-based outputs frequently exhibit:

Musical Implausibility: Monotonous pitch sequences, awkward rhythmic durations, and melodies falling outside comfortable human vocal ranges.
Misalignment: Failure to correctly map lyrics to notes.
Deployment Limitations: These flaws render the generated music unsuitable for professional use or interactive voice agents.

Existing alignment techniques like Reinforcement Learning from Human Feedback (RLHF) are effective but suffer from high computational costs and a reliance on large-scale human annotation, creating a scalability bottleneck.

2. Methodology

The authors propose a novel, fully automated alignment framework that replaces human feedback with rule-based musical constraints. The process consists of three main stages:

A. Symbolic Representation

Melodies are represented as a machine-parsable sequence of note events: $N = (l, p, d)$ , where $l$ is the lyric syllable, $p$ is the MIDI pitch, and $d$ is the duration in milliseconds. This format ensures precise alignment between lyrics and melody.

B. Rule-Based Constraint Definition

Five specific constraints are defined to detect and penalize common failure modes:

Format Constraint: Ensures the output is syntactically valid (parsable into lyric-pitch-duration tuples).
Lyric Constraint: Verifies that the generated lyrics match the input sequence (excluding melismas).
Note Constraint (Monotony Avoidance): Limits the ratio of consecutive identical pitches to prevent monotonous melodies.
Duration Constraint (Rhythmic Plausibility): Enforces valid note lengths (avoiding notes too short to hear or too long) and ensures the final note of a phrase has a conclusive, longer duration.
Register Constraint (Vocal Range): Ensures all pitches fall within a predefined human vocal range (e.g., MIDI C4 to C6).

C. Automated Preference Data Generation

Instead of human annotators, the SFT model generates multiple melody candidates for unseen lyrics. These are automatically evaluated against the five rules:

Paired Data: If a prompt yields both a rule-compliant melody ("winner") and a rule-violating melody ("loser"), a $(winner, loser)$ pair is created.
Unpaired Data: If a prompt yields only rule-violating melodies, these are collected as "undesirable" samples.

D. Sequential Alignment Strategy

The model undergoes a two-step optimization process to learn from the generated data:

Direct Preference Optimization (DPO): Applied to the paired data. This refines the model's policy to prefer compliant melodies over non-compliant ones without an explicit reward model.
Kahneman-Tversky Optimization (KTO): Applied to the unpaired negative data. This allows the model to learn from "undesirable" outputs where no "winner" existed, effectively discouraging specific failure modes (like extreme rhythmic errors) that DPO might miss.

3. Key Contributions

Rule-Based Constraint Framework: A comprehensive set of five deterministic musical rules that formalize fundamental melody writing principles, targeting specific generative failure modes.
Fully Automated Pipeline: A scalable method to generate large-scale preference datasets (both paired and unpaired) without any human intervention, eliminating the bottleneck of human annotation.
Sequential DPO-KTO Alignment: A novel strategy that combines DPO (for pairwise preference learning) and KTO (for learning from negative-only samples) to comprehensively address model weaknesses.
State-of-the-Art Performance: Demonstrated significant improvements in both objective metrics and subjective human evaluations.

4. Experimental Results

The method was evaluated on English and Chinese lyric-to-melody tasks using a test set of 1,000 sentences.

Objective Metrics:
- Pitch Distribution (PD) & Duration Distribution (DD): The proposed method achieved the highest scores (e.g., 32.37% PD and 37.11% DD on English), outperforming baselines like SongMASS, TeleMelody, and SongComposer.
- Melody Distance (MD): Achieved the lowest distance (2.63 for English), indicating closer alignment to human-composed contours.
- Rule Violations: As shown in Figure 2, the proposed method drastically reduced violations across all five constraint categories compared to the SFT baseline and ablated models.
Subjective Metrics (MOS):
- In a Mean Opinion Score test (1–5 scale) by human experts, the proposed model scored 3.42.
- This significantly outperformed all baselines (ranging from 2.92 to 3.26) and was nearly indistinguishable from Ground Truth recordings (3.50).
Ablation Studies:
- Both DPO and KTO individually improved over SFT.
- The combination (SFT + DPO + KTO) yielded the best results, confirming that DPO refines general preferences while KTO effectively penalizes specific rhythmic and structural failures found in unpaired negative data.

5. Significance

This work demonstrates that rule-based alignment is a viable, scalable, and cost-effective alternative to human feedback for instilling domain-specific knowledge into generative models. By codifying expert musical knowledge into deterministic rules, the authors successfully corrected the "musical implausibility" of LLMs without the high cost of RLHF. This approach paves the way for high-fidelity, interactive singing voice agents and automated songwriting tools that produce musically coherent and human-perceivable results.