CSyMR: Benchmarking Compositional Music Information Retrieval in Symbolic Music Reasoning

This paper introduces CSyMR-Bench, a benchmark for compositional Music Information Retrieval based on authentic user scenarios, and demonstrates that a tool-augmented framework combining ReAct-style reasoning with deterministic symbolic analysis significantly outperforms standalone Large Language Models in answering complex music score queries.

The Gist

The Big Problem: The "Music Detective" Gap

Imagine you have a giant, complex musical score (like a sheet music book for a whole orchestra) written in a special code called symbolic music. You ask a smart AI (a Large Language Model) a question about it, like: "Why does this song sound sad in the middle, and what specific chords cause that feeling?"

Current AI models are like brilliant students who have read every music theory book in the library but have never actually looked at a specific piece of sheet music.

• They can guess the answer based on what they've memorized.
• But when the answer requires looking at this specific page, counting notes, checking the rhythm, and connecting three different clues together, they often get confused or make things up (hallucinate).

The authors call this "Compositional Music Information Retrieval." In plain English: It's not just finding one fact; it's chaining together multiple small clues from the music to solve a puzzle.

The Solution: CSyMR-Bench (The "Music Exam")

To fix this, the researchers created a new test called CSyMR-Bench. Think of this as a final exam for AI music detectives.

• Where did the questions come from? They didn't just make them up. They took real questions from music students on Reddit and professional music theory exams. These are the kinds of tricky questions real humans ask.
• What makes it hard? Each question requires the AI to do a "multi-step dance." For example:
  1. Find the key of the song.
  2. Find a specific chord in measure 10.
  3. Compare that chord to the key.
  4. Conclude why it sounds "tense."
• The Scorecard: They also created a grading rubric (a taxonomy) to see where the AI fails. Did it fail because it didn't understand the rhythm? Or because it couldn't figure out the harmony? This helps them diagnose exactly what's broken.

The Magic Tool: The "Robot Librarian"

The researchers realized that asking an AI to "just think harder" wasn't working. Instead, they built a Tool-Augmented Agent.

Here is the best analogy:

• The Old Way (Pure AI): Asking a human to solve a math problem entirely from memory. If the numbers are huge, they might guess wrong.
• The New Way (Tool-Augmented): Giving that human a calculator and a ruler.

The new system works like this:

1. The Planner (The Brain): The AI reads the question and breaks it down into small steps. "Okay, first I need to find the tempo. Then I need to find the chord."
2. The Tooler (The Hands): Instead of guessing, the AI sends a command to a specialized, error-free music software (called music21). This software acts like a robot librarian that can instantly and perfectly count notes, identify chords, and check rhythms.
3. The Evidence: The robot librarian hands the AI a piece of paper with the exact facts (e.g., "Measure 5 has a C-major chord").
4. The Conclusion: The AI takes those hard facts and writes the final answer.

The Results: Why the "Robot Librarian" Wins

The researchers tested this new system against standard AI models. Here is what happened:

• Standard AI: When asked complex questions, it got about 50% right. It was guessing based on vibes.
• AI with Tools: When given the "calculator" (the music tools), it jumped to 57–60% accuracy.
• The "Aha!" Moment: The biggest improvement happened on the hardest questions—the ones that required deep analysis. The tool-based AI didn't just guess; it proved its answer.

The Metaphor:
Imagine trying to find a specific grain of sand on a beach.

• Standard AI is someone closing their eyes and pointing, hoping they get lucky.
• Tool-Augmented AI is someone who brings a metal detector, scans the beach, finds the exact spot, and then points.

Why This Matters

This paper proves that for complex tasks like music analysis, AI shouldn't try to be a "know-it-all" encyclopedia. Instead, it should be a smart manager that knows how to use reliable tools to get the facts.

By combining the AI's ability to understand human language with a robot's ability to read music code perfectly, we can build systems that don't just "sound" smart, but are actually trustworthy when analyzing art and music.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in applying Large Language Models (LLMs) to Symbolic Music Information Retrieval (MIR). While LLMs perform well on music generation and single-step theory questions, they struggle with Compositional MIR.

• The Challenge: Real-world user queries over musical scores often require multi-step reasoning where the answer is not explicitly stated but must be derived by aggregating multiple pieces of implicit evidence scattered across the score's structure.
• Limitations of Current Approaches:
  • Hallucination: Direct end-to-end reasoning by LLMs often leads to hallucinations (e.g., inventing non-existent notes) due to the "impedance mismatch" between natural language and structured symbolic notation (e.g., Humdrum *kern).
  • Benchmark Deficiencies: Existing benchmarks focus on isolated theoretical knowledge, synthetic data, or audio perception, failing to capture the complexity of authentic, multi-step retrieval tasks over symbolic scores.

2. Methodology

A. The CSyMR-Bench Benchmark

The authors introduce CSyMR-Bench, a new benchmark designed to evaluate compositional reasoning in symbolic music.

• Data Composition: Contains 126 multiple-choice questions curated from two sources:
  1. Community-Derived: Real-world inquiries from Reddit's r/musictheory (2012–2022), converted from sheet music images to symbolic Humdrum *kern format via Optical Music Recognition (OMR).
  2. Expert-Domain: Professional college-level music theory examination questions.
• Task Definition: The task is defined as identifying the correct answer $a$ from a candidate set $A$ by grounding the decision in a symbolic score $D$. This requires constructing an evidence aggregation path $R = \{e_1, e_2, ..., e_n\}$, where each $e_i$ is an atomic retrieval operation (e.g., extracting a chord root or interval) performed on the score.
• Taxonomy: To enable fine-grained diagnosis, the benchmark includes:
  • 6 Query Intent Categories: Complex Tonal-Harmonic Analysis, Editing/Rewriting, Effect/Perceptual Explanation, Composition/Creative Guidance, Complex Structural-Textural Analysis, and Genre/Musician Judgment.
  • 6 Analytical Dimensions: Pitch & Interval, Chord & Harmony, Key & Scale, Score Structural Statistics, Rhythm & Meter, and Performance & Expression.

B. Tool-Augmented Retrieval Agent

To solve the compositional retrieval problem, the authors propose a framework that integrates a ReAct-style controller with deterministic symbolic analysis operators.

• Architecture:
  • Planner: Decomposes the high-level query into actionable steps.
  • Thinker: Maintains the evidence aggregation path and dynamically formulates retrieval actions.
  • Tooler: Executes a set of 16 deterministic operators built using the music21 library.
• Key Mechanism: Instead of relying on the LLM's parametric knowledge to "guess" musical facts, the agent explicitly calls tools to perform atomic retrievals (e.g., op(D) -> e). The tools return natural language summaries of the evidence, which the LLM uses to construct the final reasoning chain.
• Design Principle: Raw code execution is hidden from the reasoning agent; the agent only processes the retrieved evidence summaries, ensuring robustness and preventing debugging distractions.

3. Key Contributions

1. CSyMR-Bench: The first benchmark dedicated to Compositional MIR in symbolic music reasoning, featuring authentic, multi-step queries grounded in real user scenarios and professional exams.
2. Fine-Grained Taxonomy: A dual-layer annotation system (Query Intents and Analytical Dimensions) that allows for detailed diagnosis of model failures beyond aggregate accuracy.
3. Tool-Augmented Framework: A novel agent architecture that treats symbolic analysis functions as structured retrieval operators. This approach significantly reduces hallucination by grounding reasoning in verifiable, deterministic evidence.

4. Experimental Results

The authors evaluated various prompting strategies (Zero-shot, Few-shot, Chain-of-Thought, ReAct) and model sizes (GPT-4.1-mini, GPT-4.1, DeepSeek, Gemini, Claude) against the proposed framework.

• Performance Gains: The Tool-Augmented Retrieval Agent (using music21 tools) achieved the highest overall accuracy (66.67%), outperforming all parametric baselines.
  • It showed a 5–7% absolute accuracy gain over the best non-tool baselines.
  • Category-Specific Results: The tool-augmented approach yielded the largest improvements on analysis-heavy categories (e.g., Complex Tonal-Harmonic Analysis: 77.42% vs. 61.29% for CoT).
  • Limitations: Improvements were negligible for metadata-driven categories (e.g., Genre Judgment), suggesting tools cannot replace the need for implicit parametric knowledge in stylistic classification.
• Model Scaling: While smaller models (GPT-4.1-mini) performed well in zero-shot settings, larger models (GPT-4.1) benefited significantly more from structured reasoning (Chain-of-Thought), reaching 69.41% in specific settings, indicating that complex compositional tasks require both strong reasoning capabilities and grounding.
• Case Study: In a key modulation detection task, a standard Chain-of-Thought baseline failed due to hallucinating non-existent pitches. The tool-augmented agent successfully detected the modulation and labeled the chord correctly by relying on deterministic tool outputs.

5. Significance

This work establishes a new paradigm for trustworthy information seeking in the musical domain. By demonstrating that grounding LLM reasoning in verifiable symbolic evidence is superior to pure probabilistic generation for complex tasks, the paper:

• Highlights the necessity of hybrid neuro-symbolic approaches for structured domains like music.
• Provides a rigorous evaluation standard (CSyMR-Bench) that moves beyond simple knowledge recall to test true compositional reasoning.
• Offers a scalable framework for integrating domain-specific tools into LLM agents, which can be adapted to other structured data domains beyond music.