⚡ electrical engineering

Exploring LLMs for South Asian Music Understanding and Generation

This paper presents the first systematic evaluation of Large Language Models on South Asian classical music, revealing that while frontier models achieve high accuracy in understanding raga-based theory, they struggle to generate stylistically faithful outputs, highlighting a significant gap in culturally grounded music modeling.

Original authors: Faria Binte Kader, Mohtasim Hadi Rafi, Shah Wasif Sajjad, Santu Karmaker

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Faria Binte Kader, Mohtasim Hadi Rafi, Shah Wasif Sajjad, Santu Karmaker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, well-read robots (Large Language Models, or LLMs) that have spent years reading books, writing code, and composing stories. They are experts at Western music—think of the piano sonatas of Beethoven or the pop songs you hear on the radio. These robots know how to build a house using Western blueprints, where the walls are made of "harmony" (chords stacking on top of each other).

But what happens if you ask these same robots to build a house using South Asian classical music blueprints?

This paper is like a rigorous inspection report. The researchers asked: Can these robots actually understand and build music based on the rules of South Asian traditions, specifically the complex systems of India and Bangladesh?

Here is the breakdown of their findings, using simple analogies:

1. The Challenge: Two Different Architectural Styles

Western music is like building with Lego bricks that snap together in specific, pre-defined patterns (harmony and chords).
South Asian music (like Hindustani classical, Rabindra Sangeet, and Nazrul Sangeet) is like weaving a tapestry. It relies on:

Raga: A specific set of "threads" (notes) and rules for how they can move, creating a specific mood.
Tala: A rhythmic cycle, like a heartbeat that repeats in a loop.
Ornamentation: Tiny, delicate flourishes (like embroidery) that happen between the notes.

The researchers wanted to see if the robots, who are used to Lego, could suddenly become master weavers.

2. The Test: A 504-Question Exam

To test the robots' "musical IQ," the researchers created a massive exam with 504 questions. It wasn't just about guessing; it tested three things:

Theory: Do they know the grammar? (e.g., "Which notes belong in this specific Raga?")
Culture: Do they know the history? (e.g., "Who composed this famous song?" or "What instrument is used?")
Continuation: If you give them the first few bars of a song, can they finish the sentence correctly?

The Results:

The Top Student: The most advanced robot (Gemini 2.5 Pro) aced the exam, scoring around 90%. It seemed to have actually "read" the books on South Asian music.
The Class Average: Most open-source robots (the free, community-built ones) scored between 23% and 40%. They were mostly guessing.
The Specialist: Interestingly, a robot specifically trained only on Western music (ChatMusician) failed miserably, scoring the lowest. This is like asking a master carpenter to fix a watch; their specific skills didn't transfer.

3. The Creation: Can They Write a Song?

Next, the researchers asked the robots to actually compose new songs in a text-based music format called "ABC notation" (think of it as a text recipe for music). They gave the robots lyrics and asked them to create a melody that sounded like a Rabindra Sangeet (songs by Tagore) or Nazrul Sangeet (songs by Nazrul Islam).

They used a "5-Level Prompt" system, which is like giving instructions with increasing detail:

Level 1: "Write a song."
Level 5: "Write a Rabindra Sangeet in this specific scale, with this rhythm, capturing this emotion, using these instruments."

The Results:
Even the best robot (Gemini 2.5 Pro) hit a wall.

Structural Validity: The robot could write a song that looked like a recipe (it had the right notes and rhythm on paper).
Stylistic Faithfulness: But when humans listened to it, only 40% of the time did it actually sound like a Rabindra Sangeet. The other 60% sounded like generic music or a different style entirely.

The Analogy: Imagine asking a robot to paint a portrait of a specific person. The robot might get the colors and the canvas right (structural validity), but the face looks like a stranger (lack of stylistic faithfulness).

4. The Trap: The "Fake" Metrics

The researchers discovered a major problem with how we usually test music AI.

The Old Way: We check if the notes are mathematically correct and if the song follows the rules of the scale.
The Reality: The robots could pass these math tests perfectly (100% score) while producing music that sounded completely wrong to a human expert.

It's like a student who memorizes the spelling of every word in a poem but has no idea how to read it with the correct emotion or rhythm. The computer says, "Great job, you spelled everything right!" but the human says, "That sounds terrible."

5. The Conclusion

The paper concludes that while the smartest AI models are starting to understand the rules of South Asian music, they are still terrible at capturing the soul of it.

Understanding: The top models are getting good at the theory (the grammar).
Generation: They are still struggling to create music that feels authentic and culturally specific.
The Future: We need better ways to test these robots. We can't just check if the notes are "correct"; we need to check if the music feels "right" to a human who knows the culture.

In short: The robots are learning the vocabulary of South Asian music, but they haven't yet learned how to speak the language with a native accent.

Technical Summary: Exploring LLMs for South Asian Music Understanding and Generation

Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in music understanding and generation, existing research and benchmarks remain heavily confined to Western tonal traditions. These works primarily evaluate capabilities based on harmonic chord progressions and Western harmony, offering little insight into whether LLMs can handle structurally distinct, low-resource musical traditions. South Asian classical music, governed by raga (melodic frameworks) and tala (cyclic rhythmic systems), operates on principles fundamentally different from Western harmony-driven composition. Specifically, the paper identifies a lack of systematic evaluation regarding:

Understanding: Whether LLMs possess the grammatical knowledge of Hindustani and Bengali classical music (e.g., raga grammar, tala systems, ornamentation).
Generation: Whether LLMs can generate symbolic music that adheres to the specific compositional conventions and stylistic constraints of South Asian genres, particularly Rabindra and Nazrul Sangeet.
Evaluation: Whether existing automatic metrics, often designed for Western music, reliably capture the structural validity and stylistic faithfulness required for culturally grounded music modeling.

Methodology

1. Benchmark Curation (Music Understanding)

The authors constructed a 504-question multiple-choice benchmark grounded in Hindustani classical theory and Bengali classical traditions. The benchmark is divided into three subtasks:

Music Theory Understanding (163 questions): Evaluates comprehension of theoretical grammar, including Raga grammar, Tala systems, Thaat classification, ornamentation (gamakas), and notation.
Music General Knowledge (143 questions): Assesses factual and cultural knowledge regarding composers, historical figures, instruments, and regional stylistic traditions.
Music Continuation (198 questions): Tests practical application of structural knowledge. Models are presented with the beginning of a Rabindra or Nazrul Sangeet in ABC notation and must select the most structurally coherent continuation from four options. Distractors were constructed by introducing structural mismatches (e.g., cross-genre substitution, rhythmic inconsistencies) while maintaining plausible surface-level continuity.

All questions were curated in Bengali to preserve terminology and reviewed by an expert in Bengali classical music.

2. Music Generation Framework

To evaluate generation capabilities, the authors designed a 5-level controlled prompting framework based on the TELeR taxonomy. This framework progressively introduces musical constraints (scale, rhythm, genre, stylistic cues) to analyze controllable symbolic music generation.

Data: 100 manually curated reference scores (50 Rabindra Sangeet, 50 Nazrul Sangeet) in ABC notation, derived from official Swaralipi sources, served as seeds and references.
Prompting: For 100 lyrics, five prompt variants were generated per model, totaling 500 prompts. The prompts ranged from minimal details (Level 1) to maximum detail including background on Indian classical music principles (Level 5).
Models Evaluated: 33 LLMs (30 open-source, 3 proprietary) were tested on understanding. The top 9 performers were selected for generation evaluation.

3. Evaluation Metrics

Automatic Evaluation:
- Reference-Based: KL Divergence between the pitch distributions of generated and reference scores.
- Reference-Free: Adherence to scale, ABC syntax accuracy (parse success rate), repetition rate, and pitch histogram entropy.
Human Evaluation: Three annotators with formal music theory training evaluated 180 samples (20 per top model) generated using Level 3 prompts. Ratings were provided on six criteria: Structureness, Genre analysis, Style analysis, Emotion, Adherence to instruction, and Harmoniousness. Inter-Annotator Agreement (IAA) was calculated using Krippendorff's $\alpha$ and Cohen's/Fleiss' $\kappa$ .

Key Contributions

First Systematic Benchmark: Introduction of a 504-question benchmark spanning music theory, general knowledge, and symbolic continuation, serving as the first foundational diagnostic for LLM competence in South Asian classical music.
Curated Dataset: Manual curation of 100 reference scores in ABC notation for Rabindra and Nazrul Sangeet, publicly released for future research.
Controlled Prompting Framework: Proposal of a 5-level prompting framework to systematically analyze the impact of musical constraints on symbolic generation.
Comprehensive Evaluation: Evaluation of 33 LLMs on understanding and the top 9 on generation, revealing significant capability gaps and the limitations of current automatic metrics.

Results

Music Understanding

Performance Gap: Frontier models significantly outperformed open-source models. Gemini 2.5 Pro achieved the highest accuracy across all subtasks (90.8% Theory, 89.4% Knowledge, 85.2% Continuation). In contrast, most open-source models scored in the 23–40% range.
Scaling Limits: Within model families (e.g., Qwen2.5), increasing parameter size yielded diminishing returns, with 32B, 14B, and 7B variants scoring nearly identically on theory understanding.
Reasoning vs. Knowledge: Reasoning-focused models (e.g., QwQ-32B-Preview) did not outperform general models on theory understanding, indicating that chain-of-thought reasoning cannot compensate for a lack of culturally grounded musical knowledge.
Specialized Models: ChatMusician, a model fine-tuned on Western ABC notation, performed poorly (12.3% Theory), confirming that fine-tuning on Western traditions does not transfer to South Asian styles.

Music Generation

Structural vs. Stylistic Faithfulness: Even the strongest model (Gemini 2.5 Pro) produced outputs that were stylistically faithful (recognizable as Rabindra or Nazrul Sangeet) in only 40% of cases, despite achieving 95% adherence to the broader South Asian genre.
Prompt Sensitivity: Levels 2 and 3 of the prompting framework yielded competitive scores. Levels 4 and 5, despite adding more context, did not significantly improve generation quality, suggesting a ceiling in current model capabilities for stylistic control.
Metric Limitations: Automatic metrics showed weak correlation with human judgment. Only KL Divergence showed a statistically significant correlation with human style accuracy ( $r = -0.73$ ). Metrics like Scale Adherence and ABC Syntax Accuracy showed near-zero correlation with style accuracy, indicating that syntactic correctness does not imply stylistic faithfulness.
Human Evaluation: Most open-source models scored below 2.5/5 on ordinal criteria (Structureness, Emotion, etc.), indicating that structurally parseable outputs do not translate to perceptually coherent or emotionally aligned melodies.

Significance and Claims

The paper claims that while frontier LLMs show emergent comprehension of South Asian musical knowledge, there remains a critical gap between structural generation competence and fine-grained stylistic faithfulness. The authors assert that:

Cultural Grounding is Missing: Current LLMs can approximate surface characteristics of South Asian music but lack the compositional specificity required for faithful stylistic control.
Evaluation Deficiencies: Existing automatic metrics are insufficient for culturally grounded evaluation. They operate at the syntactic and pitch-distribution level and can be satisfied trivially while producing outputs that are stylistically unrecognizable to human experts.
Open Challenge: Developing metrics sensitive to raga-specific ornamentation and creating culturally grounded evaluation frameworks that move beyond syntactic correctness remain open problems for the field.

The study concludes that achieving faithful stylistic control in low-resource musical traditions requires more than just scaling current models; it necessitates new evaluation frameworks and potentially new modeling approaches that account for the unique structural principles of non-Western music.