← Latest papers

💬 NLP

A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

This paper introduces a new sentence-level song lyric dataset to analyze human-LLM alignment in emotion recognition and proposes a hybrid annotation framework that optimizes the process by predicting potential misalignments between human and model annotations.

Original authors: Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan, Aditya Joshi, Erik Meijering

Published 2026-06-30

📖 4 min read☕ Coffee break read

Original authors: Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan, Aditya Joshi, Erik Meijering

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand the emotional story of a song. Usually, we just listen to the music and guess how it feels. But this paper argues that the words (lyrics) tell their own, often different, story that changes from line to line. The problem is, labeling these emotions is tricky, expensive, and confusing because different people feel things differently.

Here is a simple breakdown of what the researchers did, using some everyday analogies.

The Problem: The "Human vs. Robot" Emotion Debate

Think of annotating song lyrics like trying to describe the weather in a specific city.

Humans are like local residents. They know the culture, the slang, and the subtle mood of the neighborhood. But, they might disagree. One person might say, "It's a gloomy Tuesday," while another says, "It's just a quiet Tuesday." They are sensitive but inconsistent.
Large Language Models (LLMs) are like super-fast weather satellites. They can scan thousands of cities in seconds and give a very consistent report. However, they might miss the local nuance. They might see a "cloud" and label it "rain," missing the fact that the locals call it "drizzle" and it feels cozy, not sad.

The researchers wanted to know: Can we get the best of both worlds? Can we use the robot's speed and the human's heart to label emotions in songs?

Step 1: The Experiment (The "Taste Test")

The team created a dataset of 652 lines of lyrics from 50 songs (ranging from pop to classical). They asked two groups to label the emotion of each line using two scales:

Valence: How positive or negative is it? (Happy to Sad)
Arousal: How energetic is it? (Calm to Excited)

The Results:

Humans were great at catching subtle, poetic, or cultural meanings, but they disagreed with each other a lot, especially on how "energetic" a line felt.
LLMs (like GPT-4, Gemini, and LLaMA) were very consistent with themselves. If they thought a line was "calm," they usually agreed on that. However, they sometimes missed the deep, metaphorical sadness or joy that humans caught immediately.

The Analogy:
If the lyric was "I'm so happy," everyone agreed. But if the lyric was a complex metaphor like "My heart is a broken clock," humans argued about what that meant, while the robots gave a very consistent, but perhaps slightly "off," answer.

Step 2: The Solution (The "Smart Traffic Light")

Instead of choosing either humans or robots, the researchers built a Hybrid Framework. Think of this as a smart traffic light system for data.

The Predictor (The Traffic Light): Before a line of lyrics gets labeled, a small AI program looks at it. It asks: "Is this line simple and straightforward? Or is it complex, metaphorical, and tricky?"
The Routing:
- If the line is simple (e.g., "The sun is shining"), the system sends it to the LLM. It's fast, cheap, and the robot is good enough.
- If the line is complex (e.g., "I'm drowning in a sea of silence"), the system sends it to a Human. The robot might get confused by the metaphor, so a human is needed to interpret the feeling.
The Aggregation (The Final Score): When multiple people or robots label the same line, the system doesn't just take an average. It gives more "voting power" to the ones who have been proven to be reliable in the past.

Step 3: Did it Save Money?

Yes, significantly.

Human-only approach: Imagine hiring 10 people to read 42,000 lines of lyrics. It would take months and cost roughly $87,500.
Hybrid approach: By letting the robots do the easy 74% of the work and only sending the tricky 26% to humans, the cost drops to under $75 (in API fees) plus a small amount for human verification.

The Catch:
This system only starts saving money once you have a lot of data (more than 1,000 lines). If you only have a few songs, it's actually faster to just have a human do it all.

The Bottom Line

The paper concludes that we shouldn't try to replace humans with AI, nor should we ignore AI. Instead, we should build a team.

Use AI for the heavy lifting and the straightforward stuff.
Use Humans for the tricky, artistic, and culturally deep parts.

By combining them, you get a system that is fast, cheap, and accurate, capturing the true emotional evolution of a song line-by-line without breaking the bank.

Technical Summary: A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

1. Problem Statement

Emotion recognition in music (MER) has traditionally relied on audio features, often treating lyrics as secondary or assigning them the same emotional label as the full song. This approach overlooks the fact that lyrics possess unique, evolving emotional meanings that may not align with the audio, varying dynamically line-by-line. Furthermore, annotating lyrics is inherently challenging due to:

Subjectivity and Variability: Inter-annotator disagreement driven by cultural background, individual interpretation, and linguistic nuance.
Scalability: Manual annotation is labor-intensive and costly, particularly at the sentence level required to capture temporal emotional shifts.
Ambiguity: Metaphorical and context-dependent lyrics often lack clear textual cues for consistent emotional labeling.

The paper investigates the alignment between humans and Large Language Models (LLMs) for sentence-level lyric emotion annotation and proposes a hybrid framework to leverage the complementary strengths of both.

2. Methodology

A. Dataset Curation

The authors constructed a new sentence-level dataset comprising 652 lyric segments from 50 songs (selected from the DEAM and PMEmo datasets).

Granularity: Lyrics were manually segmented into semantically meaningful sentence-level units to capture dynamic emotional shifts.
Labels: Each segment was annotated for Valence and Arousal (VA) on a continuous scale from -1 to 1, aligning with the Russell circumplex model.
Diversity: The dataset covers various genres (pop, rock, classical, jazz, electronic) and spans transnational creative influences.

B. Annotation Procedure

Two distinct annotation pipelines were executed independently:

Human Annotation: Three PhD students (East/South Asian background) annotated segments using a custom HTML tool with a VA circumplex plot.
LLM Annotation: Three state-of-the-art models (GPT-4, Gemini 1.5, LLaMA 3) were prompted using a "Semantic Anchoring" approach. The prompts provided 28 reference (Valence, Arousal) coordinates for common emotions to standardize the output space without forcing discrete categories.

C. Comparative Analysis

The study analyzed agreement and correlation:

Consistency: LLMs showed higher internal consistency for Arousal (Mean Std: 0.182 vs. Human 0.215) but slightly higher variability for Valence (Mean Std: 0.251 vs. Human 0.233).
Length Dependency: Human disagreement in arousal increased with lyric length, whereas LLM disagreement in valence decreased as context increased.
Correlation: Human-LLM correlations were generally low (e.g., $r=0.29$ for Valence), indicating a mismatch in affective polarity judgments, while LLM-LLM correlations were stronger.

D. The Hybrid Annotation Framework

To optimize quality and efficiency, the authors propose a three-component framework:

Weighted Aggregation:
- Human Weights: Calculated based on pairwise Kendall's Tau correlations among human annotators.
- LLM Weights: Calculated based on Spearman's rank correlation between each LLM's output and the aggregated human consensus.
- Result: A reliability-weighted score that smooths annotator noise.
Annotation Source Prediction:
- A supervised learning approach to predict the optimal annotator (Human vs. specific LLM) for each lyric line.
- Features: Linguistic complexity (length, syntax, figurativeness), semantic features (sentiment polarity), disagreement indicators, and LLM confidence signals.
- Proxy Labels: Since ground truth for the "best" annotator is unavailable, a proxy label was constructed using weighted consensus, prioritizing sources with lower deviation and better handling of figurative content.
- Models Evaluated: Decision Tree, Random Forest, Semi-Supervised Learning, and Reinforcement Learning. The Random Forest achieved the highest accuracy (67.0%), particularly in identifying lines suitable for LLM annotation (85% recall).
Routing Logic:
- The framework dynamically routes segments: simple or literal lines are assigned to LLMs, while complex, metaphorical, or ambiguous lines are routed to human experts.
- Ablation Study: Identified Linguistic Complexity and Annotator Agreement as the primary drivers for routing decisions, accounting for nearly half of the dataset's routing logic.

3. Key Results

Complementarity: Humans excel in interpreting subtle, metaphorical, and culturally nuanced lyrics but suffer from high variability. LLMs provide consistent, scalable annotations for straightforward text but struggle with subjective valence in complex contexts.
Cost Efficiency:
- Human-only: Estimated at A$87,500 for the full dataset (approx. 42,000 lines).
- LLM-only: Estimated at A$75 (API fees) or GPU costs for local inference.
- Hybrid: By routing ~26% of lines (high complexity) to humans and ~74% to LLMs, the framework reduces human labor costs by an estimated 70% while maintaining high-quality annotation.
Performance: The source prediction models achieved accuracies between 62% and 67%, sufficient to serve as a "first-pass" filter to optimize resource allocation.
Iterative Optimization: A pilot experiment fine-tuning LLaMA on hybrid-annotated segments resulted in a 15.4% reduction in routing triggers, suggesting the system can improve over time by internalizing human expert logic.

4. Significance and Claims

The paper claims that the proposed hybrid framework offers a robust and scalable solution for lyric emotion annotation by:

Bridging the Gap: It addresses the subjectivity of human annotation and the lack of nuance in automated annotation by dynamically selecting the best source based on lyric characteristics.
Cost-Effectiveness: It demonstrates that significant cost reductions (up to 70%) are achievable without compromising annotation quality, provided the dataset scale is sufficient (breaking even at ~1,000 lines).
Methodological Contribution: It introduces a novel "Semantic Anchoring" prompt strategy for LLMs and a weighted aggregation method that leverages inter-annotator reliability to create high-quality proxy labels for training.
Future Direction: The work lays the foundation for more accurate multimodal MER systems by disentangling the emotional contributions of lyrics from audio, moving beyond song-level labels to temporally dynamic, sentence-level analysis.

The authors remain modest, acknowledging limitations such as the small, culturally homogeneous human annotator pool and the moderate accuracy of the prediction models. They suggest that future work should focus on integrating audio features, expanding to multilingual datasets, and refining the routing logic with deeper contextual embeddings.