Investigating Modality Contribution in Audio LLMs for Music

Technical Summary: Investigating Modality Contribution in Audio LLMs for Music

Problem Statement
Audio Large Language Models (Audio LLMs) are designed to facilitate human-like conversation about music by integrating audio information into reasoning processes. However, recent benchmarks, such as MuChoMusic, reveal a paradox: while these models achieve high accuracy on multiple-choice questions, indirect experiments (e.g., replacing audio with silence or noise) suggest they may not be genuinely "listening." Instead, they might rely predominantly on textual reasoning or prior knowledge, a phenomenon known as modality imbalance or unimodal collapse. Current evaluation methods relying on accuracy metrics are insufficient because they fail to distinguish between models that correctly use audio and those that arrive at correct answers via text alone. There is a critical need for a performance-agnostic method to quantify the specific contribution of each modality (audio vs. text) to a model's prediction.

Methodology
To address this, the authors adapt the MultiModal-SHAP (MM-SHAP) framework, originally proposed for vision-language models, to the domain of Audio LLMs. The core methodology relies on Shapley values from cooperative game theory to measure the marginal contribution of input features to a model's output.

1. Feature Perturbation via Masking: The study employs a masking strategy to approximate Shapley values.
   • Audio: Raw waveform segments are zeroed out. The masking window size is dynamically adjusted based on the number of text tokens to ensure a balanced feature count between modalities ($n_A \approx n_T$).
   • Text: Specific tokens are replaced with the [MASK] token. Special tokens (e.g., audio indicators, question markers) are preserved to maintain structural integrity.
2. Score Computation: Unlike classification tasks where probabilities are measured, this study addresses generation tasks. The authors identify the tokens corresponding to the model's generated answer (e.g., "Doorbell") in an unmasked baseline run. They then compute the change in logits for these specific tokens when subsets of audio and text features are masked.
3. Modality Contribution Metrics: The study aggregates the absolute Shapley values for all audio features ($\Phi_A$) and text features ($\Phi_T$). To enable direct comparison, they define normalized ratios:
   • A-SHAP: $\frac{\Phi_A}{\Phi_T + \Phi_A}$
   • T-SHAP: $1 - \text{A-SHAP}$
     A perfectly balanced model would yield an A-SHAP of 0.5.

Experimental Setup
The authors evaluated two state-of-the-art Audio LLMs on the MuChoMusic benchmark:

• MU-LLaMA: Uses MERT-v1-330M as the audio encoder and LLaMA 2 7B as the language model.
• Qwen-Audio: Uses Whisper-large-v2 as the audio encoder and Qwen-7B as the LLM.

Experiments were conducted under two conditions: Multiple-Choice with Previous Instructions (MC-PI) and Multiple-Choice with No Previous Instructions (MC-NPI). The study also included a qualitative analysis of specific questions requiring the localization of single sound events versus long-term audio understanding.

Key Results

• Inverse Relationship between Accuracy and Audio Usage: The model with higher accuracy (Qwen-Audio) relied significantly less on the audio modality (A-SHAP $\approx$ 0.21–0.23) compared to the less accurate model (MU-LLaMA, A-SHAP $\approx$ 0.47–0.50). This indicates that higher performance on MuChoMusic does not correlate with balanced modality usage.
• Text Dominance in Multiple-Choice: Both models exhibited a strong bias toward text, particularly in the MC-PI setup, suggesting that the multiple-choice format allows models to solve problems using textual reasoning or distractor elimination rather than perceptual audio analysis.
• Audio is Not Ignored: Despite low overall A-SHAP scores, qualitative inspection revealed that audio features still contribute meaningfully. For tokens corresponding to ground-truth sound events (e.g., "bell"), positive Shapley values aligned with the specific time segments containing those sounds. However, the magnitude of audio contributions was often lower than text, and in some cases, strong activations occurred in regions unrelated to the correct answer (e.g., piano notes when the answer was a bell).
• Task Dependence: When the task was shifted from multiple-choice to open-ended audio description, the A-SHAP for Qwen-Audio increased significantly (to 0.73), demonstrating that modality usage is highly dependent on the task formulation.

Significance and Claims
The paper claims to be the first application of MM-SHAP to Audio LLMs. Its primary contribution is providing a quantitative, performance-agnostic framework to inspect how these models utilize audio versus text. The authors argue that their findings challenge the assumption that optimal multimodal models must utilize all modalities equally; a model can achieve high accuracy while relying heavily on a single modality.

The study highlights a gap between "listening" (perceptual processing) and "reasoning" (textual inference). While the models are not entirely ignoring audio, the low contribution scores in multiple-choice tasks suggest that current benchmarks may be testing an LLM's ability to eliminate distractors rather than its ability to perceive sound. The authors conclude that MM-SHAP serves as a foundational step for Explainable AI (XAI) in audio, though they acknowledge limitations regarding the lack of ground truth for complex musical concepts (melody, structure) and the sensitivity of results to masking strategies and window sizes. Future work is proposed to systematically study how audio information survives multimodal integration across different task formulations.