Technical Summary: DeRA-MOS

Problem Statement

The evaluation of Text-to-Music (TTM) systems remains a bottleneck due to the high cost and subjectivity of human Mean Opinion Scores (MOS) for Music Impression (MI) and Text Alignment (TA). While automatic MOS estimators exist, they typically rely on point-wise regression (e.g., L1, MSE) or distributional classification objectives. These approaches suffer from two primary limitations:

1. Training-Evaluation Mismatch: Standard point-wise objectives treat samples independently, failing to directly optimize rank-based metrics like Spearman's Rank Correlation Coefficient (SRCC), which are the standard for evaluating relative system quality.
2. Modality Drift: In TA estimation, cross-attention mechanisms often lack explicit geometric constraints on unimodal embeddings prior to fusion. Without anchoring to human judgment, these representations can drift, leading to poor semantic coherence and memorization of training data rather than true alignment.

Methodology

The authors propose DeRA-MOS (Decoupled Ranking and Alignment), a framework that decouples the optimization objectives for MI and TA while maintaining a shared backbone architecture (based on the DORA-MOS dual-branch structure). The framework introduces two specialized, training-only loss functions:

1. Batch-Aware Listwise Ranking (BALR) for Music Impression

To address the mismatch between point-wise training and rank-based evaluation, the MI branch replaces standard point-wise supervision with a listwise ranking loss.

• Mechanism: The entire mini-batch is treated as a single query list. Ground-truth and predicted MI scores are converted into probability distributions using a temperature-scaled softmax.
• Objective: The loss minimizes the cross-entropy between the target and predicted probability distributions.
• Benefit: This approach optimizes the global ordering within a batch, providing gradients consistent with SRCC. It captures global batch dynamics with $O(B)$ complexity, avoiding the $O(B^2)$ cost of pairwise methods while respecting the relative nature of human ratings.

2. Score-Anchored Modality Alignment (SAMA) for Text Alignment

To prevent representation drift in the TA branch, a geometric constraint is applied before the cross-attention fusion step.

• Mechanism: The method computes the cosine similarity between pre-fusion audio and text embeddings (L2-normalized). This similarity is linearly mapped to the [0, 1] range.
• Objective: A regression loss is applied to align this mapped similarity directly with the normalized human TA MOS scores.
• Benefit: This "anchors" the latent audio-text similarity to human judgment, regularizing the latent space and ensuring that high-scoring pairs are geometrically closer before the network attempts to fuse them via attention.

Joint Optimization

The total loss function combines the baseline Gaussian-softened classification losses ($L_{CE-Gauss}$) with the proposed auxiliary losses:
$$L_{Total} = L_{CE-Gauss} + \alpha L_{BALR} + \beta L_{SAMA}$$
Crucially, these auxiliary losses are applied only during training. During inference, the framework operates with the exact same complexity and parameters as the baseline, ensuring no additional computational overhead.

Key Contributions

1. Batch-Aware Listwise Ranking: The introduction of a listwise loss that directly aligns MI optimization with SRCC evaluation metrics, moving beyond independent point-wise predictions.
2. Score-Anchored Modality Alignment: A novel loss that enforces geometric consistency between latent audio-text similarity and human TA scores, mitigating representation drift.
3. Robust Decoupled Framework: The demonstration that combining these decoupled objectives yields superior performance in both ranking (SRCC, Kendall's Tau) and absolute accuracy (MSE) compared to homogeneous point-wise baselines, without increasing inference cost.

Experimental Results

Evaluated on the MusicEval dataset (the official benchmark for the AudioMOS 2025 Challenge), DeRA-MOS demonstrated significant improvements over a reproduced DORA-MOS baseline:

• Music Impression (MI): The full model achieved an SRCC of 0.989 and a Kendall's Tau (KTAU) of 0.940, improving KTAU by +0.050 over the baseline. This indicates a substantial reduction in rank inversions.
• Text Alignment (TA): The model improved SRCC from 0.952 to 0.956 and significantly reduced MSE from 0.060 to 0.028, demonstrating better calibration of absolute scores.
• Statistical Significance: Paired Wilcoxon signed-rank tests confirmed that the gains in SRCC for both tasks were statistically significant ($p < 0.01$).
• Ablation Findings:
  • Using only BALR improved MI ranking but degraded TA performance, confirming that ranking pressure alone can distort shared representations.
  • Using only SAMA sharply improved TA calibration while preserving ranking metrics.
  • The combination of both losses yielded the best overall profile, confirming their complementary nature.
• Hyperparameter Sensitivity: Optimal performance was observed with a batch size of 32 and a temperature scaling factor ($\tau$) of 1.0. The framework remained robust across varying loss weighting coefficients ($\alpha, \beta$).

Significance and Claims

The paper claims that DeRA-MOS establishes a robust paradigm for large-scale TTM evaluation by effectively bridging the gap between training objectives and evaluation metrics. By decoupling the optimization of ranking (MI) and alignment (TA), the framework resolves the specific bottlenecks of point-wise training mismatch and modality drift.

The authors emphasize that because the proposed losses are training-only regularizers, the framework offers substantial performance gains without compromising inference efficiency. This makes DeRA-MOS particularly suitable for the iterative development and evaluation of large-scale generative music models where computational resources and evaluation costs are critical constraints. The work suggests that explicit geometric anchoring and listwise ranking are essential components for accurate, scalable automatic MOS estimation.