Beyond Musical Descriptors: Extracting Preference-Bearing Intent in Music Queries

1. Problem Statement

While annotated datasets for extracting musical descriptors (e.g., genre, mood, artist) from user queries are becoming common, existing approaches fail to capture the user's intent behind these descriptors.

• The Gap: Current systems treat descriptors as static entities. However, a descriptor can express a positive affinity (e.g., "I want Rock"), a negative aversion (e.g., "No Elvis"), or a referential similarity (e.g., "Like Elvis but newer").
• The Challenge: Large Language Models (LLMs) struggle to distinguish between these nuanced roles, particularly with context-dependent descriptors, negation, and softer similarity concepts. Existing datasets do not explicitly label these "preference-bearing intents."

2. Methodology

A. Dataset Construction: MusicRecoIntent

The authors introduced MusicRecoIntent, a manually annotated corpus derived from the MusicRecoNER dataset (Reddit music requests).

• Scale: 2,291 user queries containing 3,935 total annotations.
• Annotation Schema:
  • Categories: 7 descriptor types: Decade, Genre, Instrument, Listening Context (LC), Mood, Named Entities (NE), and Country.
  • Preference Roles: Each descriptor is labeled with one of three intents:
    • Positive (+): Desired attribute.
    • Negative (-): Undesired attribute (avoidance).
    • Referential (~): Used as a similarity anchor or reference point.
• Annotation Process: Two annotators labeled descriptors and intents. Inter-annotator agreement was high for intent (Cohen's Kappa = 0.927) and substantial for extraction (Global agreement = 0.771). Disagreements often stemmed from segmentation boundaries (e.g., "Frank Ocean" vs. "Frank Ocean, Blonde") or normalization issues.

B. Experimental Setup

The study evaluated the ability of various Large Language Models (LLMs) to perform two sequential tasks:

1. Descriptor Extraction: Identifying musical entities and attributes.
2. Intent Classification: Assigning the correct preference role (+, -, ~) to each extracted descriptor.

• Models Tested: A diverse range of open-source models via Ollama, including Gemma 3 (4B, 12B, 27B), LLaMA 3 (1.8B/8B), Mistral (7B), and Qwen 3 (8B, 32B).
• Prompting Strategy:
  • Prompt 1: Instructed models to identify entities using a broad definition, supported by six concrete examples (few-shot learning).
  • Prompt 2: A subsequent prompt to determine the preference-bearing intent for the extracted descriptors.
• Evaluation Metrics:
  • Extraction: Precision, Recall, and F1-score calculated under two conditions: Exact Match (strict segmentation) and Partial Match (at least one word overlap).
  • Intent: Confusion matrix analysis and accuracy on the subset of descriptors correctly extracted by the model.

3. Key Contributions

1. MusicRecoIntent Corpus: The first dataset to explicitly link musical descriptors with their preference-bearing roles (positive, negative, referential), enabling fine-grained intent modeling.
2. LLM Benchmark: A comprehensive evaluation of state-of-the-art LLMs on the dual task of descriptor extraction and intent classification in the music domain.
3. Error Analysis: A qualitative analysis revealing that errors arise not just from model limitations but also from inherent ambiguities in human annotation guidelines (e.g., segmentation boundaries, granularity).

4. Key Results

Descriptor Extraction Performance

• Top Performers: Gemma 3 (27B) and Qwen 3 (8B) achieved the highest performance.
  • Exact Match F1: 0.69 (Gemma 27B).
  • Partial Match F1: 0.76 (Gemma 27B).
• Category Performance:
  • High Accuracy: Named Entities (NE), Country, Decade, and Genre were extracted with high precision (F1 > 0.80).
  • Lower Accuracy: Listening Context (LC) and Instrument had lower scores (F1 ~0.45–0.66). This was attributed to high variability in human annotation boundaries (e.g., whether "trip to Norway" is one context or two) rather than solely model failure.
• Observation: Models generally extracted more descriptors than the human baseline, leading to a need for strict filtering to avoid hallucinations.

Preference-Bearing Intent Prediction

• Overall Accuracy: On the subset of correctly extracted descriptors, the model achieved 89% accuracy in predicting the intent.
• Bias Analysis:
  • Positive (+): Predicted with high accuracy (94% recall).
  • Negative (-): High accuracy (90% recall), though rare in the dataset.
  • Referential (~): The model struggled most here, frequently misclassifying referential intents as positive (+). Specifically, 231 cases were incorrectly predicted as positive, often involving Named Entities (e.g., interpreting "like Elvis" as a desire for Elvis rather than a similarity anchor).
• Negation Handling: While models handle explicit negation well, they struggle with complex sentences where a negative clause modifies a positive overall intent (e.g., "I hate rap but want to get into the genre").

5. Significance and Conclusion

• Benchmarking: MusicRecoIntent serves as a critical benchmark for developing next-generation music recommendation systems that understand why a user wants a song, not just what they want.
• LLM Capabilities: The study confirms that LLMs are robust at extracting explicit, well-defined musical attributes but require further refinement for context-dependent descriptors and nuanced intent (specifically referential similarity).
• Future Directions: The authors highlight that improving these systems requires:
  1. Clearer Annotation Guidelines: To resolve human ambiguity regarding segmentation and granularity.
  2. Prompt Engineering: To better guide LLMs in distinguishing between "liking" an entity and using it as a "reference."
  3. Handling Ambiguity: Developing models that can resolve genuine semantic ambiguities where multiple interpretations are plausible.

In summary, this work moves beyond simple entity extraction to a more sophisticated understanding of user intent, providing the necessary data and benchmarks to build more intuitive and accurate music search and recommendation engines.