ArtistMus: A Globally Diverse, Artist-Centric Benchmark for Retrieval-Augmented Music Question Answering

1. Problem Statement

Despite the advancements of Large Language Models (LLMs) in open-domain question answering, their performance in Music Question Answering (MQA) remains suboptimal. Key challenges include:

• Sparse Domain Knowledge: Pretraining data often lacks sufficient depth in music-specific facts, leading to hallucinations regarding artist biographies, discographies, and historical contexts.
• Limitations of Fine-tuning: Traditional domain adaptation via fine-tuning is computationally expensive, requires high-quality training data, and struggles to incorporate dynamic updates (e.g., new album releases or career milestones) without retraining.
• Lack of Specialized Resources: Existing benchmarks focus heavily on music theory, multimodal understanding, or general knowledge, neglecting artist-centric metadata (genre, debut year, collaborations) which is crucial for everyday music discovery.
• Inadequate Retrieval: General knowledge bases like Wikipedia lack the density and structure required for efficient music retrieval, and no dedicated music-specific vector database exists for Retrieval-Augmented Generation (RAG).

2. Methodology

The authors propose a two-pronged approach involving the creation of a specialized knowledge base and a rigorous benchmark, followed by systematic evaluation of RAG strategies.

A. MusWikiDB (Specialized Knowledge Base)

• Construction: A vector database constructed from 144,000 music-related Wikipedia pages, yielding 3.2 million passages.
• Scope: Covers seven categories: Artists, Genres, Instruments, History, Technology, Theory, and Forms.
• Processing:
  • Content was crawled up to a depth of 3 from root pages.
  • Text was segmented into passages of up to 256 tokens with a 10% overlap to preserve context.
  • Short sections (<60 tokens) were removed.
  • Retrieval: Uses BM25 for first-stage retrieval and BGE-reranker for second-stage refinement.
• Optimization: Unlike the general Wikipedia corpus, MusWikiDB is domain-specific, resulting in higher retrieval relevance and faster inference speeds.

B. ArtistMus (Benchmark Dataset)

• Composition: A benchmark of 1,000 multiple-choice questions covering 500 diverse artists from 163 countries.
• Diversity Strategy:
  • Geographic: Prioritizes underrepresented regions (e.g., Mauritius, Senegal) using an inverse frequency strategy, ensuring ~43% of artists are from outside the US/Europe.
  • Metadata: Includes genre, debut year, and topic categories.
  • Question Types:
    • Factual: Verifiable details (e.g., "What vocal range does Robyn have?").
    • Contextual: Requires reasoning over narrative information (e.g., "How did Robyn's approach change while writing Honey?").
• Validation: A two-stage process involving LLM verification (Music Relevance and Faithfulness) and human validation to ensure high quality.

C. Experimental Setup

• Models: Evaluated various LLMs including closed-source (GPT-4o, Gemini 2.5 Flash) and open-source (Llama 3, Qwen3, ChatMusician) ranging from 0.6B to 8B parameters.
• Settings:
  1. Zero-shot: Baseline performance without retrieval.
  2. RAG: Retrieval using MusWikiDB (BM25) + Generation.
  3. Rerank: Two-stage retrieval (BM25 + BGE Reranker).
  4. Fine-tuning: Compared standard QA fine-tuning vs. RAG-style fine-tuning (training on context-question-answer triples).
• Benchmarks: Evaluated on ArtistMus (in-domain) and TrustMus (out-of-domain) to test generalizability.

3. Key Contributions

1. MusWikiDB: The first large-scale, music-specific vector database (3.2M passages) optimized for RAG, offering higher retrieval accuracy and 40% faster speed compared to general Wikipedia.
2. ArtistMus: A globally diverse benchmark focusing on artist metadata, filling the gap in artist-centric MQA evaluation.
3. Empirical Validation of RAG in Music: Demonstrated that RAG significantly bridges the performance gap between open-source and proprietary models in music domains.
4. RAG-Style Fine-tuning: Showed that training models to utilize retrieved context improves both factual recall and contextual reasoning, outperforming standard QA fine-tuning.

4. Results

• Factual Accuracy: RAG dramatically improved factual accuracy. For example, Qwen3 8B improved from 35.0% (Zero-shot) to 91.8% (RAG + Rerank), a gain of +56.8 percentage points.
• Proprietary vs. Open-Source: With retrieval, smaller open-source models (e.g., Llama 3.1 3B) approached or matched the performance of large proprietary models (GPT-4o, Gemini).
• Contextual Reasoning: While factual gains were massive, contextual reasoning saw more modest but significant improvements (+4.8 pp on average), suggesting retrieval provides the necessary facts while the model handles the reasoning.
• Fine-tuning Insights:
  • Standard QA fine-tuning slightly improved facts but degraded contextual performance (likely due to overfitting).
  • RAG-style fine-tuning yielded the best overall results (93.1% accuracy), proving that teaching models to integrate external evidence is superior to memorization.
• Out-of-Domain Transfer: The approach generalized well to the TrustMus benchmark, confirming that MusWikiDB enhances retrieval capabilities even for knowledge sources not explicitly in the training database.
• Retriever Configuration: BM25 outperformed dense embedding models (Contriever) for this entity-centric task. The optimal configuration was 256-token passages with a reranker.

5. Significance and Impact

• Democratization of Music QA: The study proves that small, open-source models equipped with domain-specific retrieval can rival expensive, closed-source systems, making high-quality music QA accessible and cost-effective.
• Dynamic Knowledge: The RAG framework solves the "static knowledge" problem of LLMs, allowing systems to access up-to-date information without costly retraining.
• Cultural Representation: By prioritizing artists from 163 countries, ArtistMus challenges the Western-centric bias prevalent in existing music datasets and LLM training data.
• Foundation for Future Research: The release of MusWikiDB and ArtistMus establishes a new standard for evaluating retrieval-augmented reasoning in culturally rich, knowledge-intensive domains beyond music.

Conclusion: The paper establishes that Retrieval-Augmented Generation is the most effective strategy for Music Question Answering, particularly when paired with a specialized, diverse knowledge base like MusWikiDB. This approach not only fixes hallucinations but also enables smaller models to perform at state-of-the-art levels.