Technical Summary: FuScore – Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment

1. Problem Statement

Infrared-Visible Image Fusion (IVIF) aims to integrate thermal information and detailed spatial structures into a single fused image. However, evaluating the quality of these fused images remains a significant challenge. Existing evaluation approaches suffer from two primary limitations:

1. Misalignment with Human Perception: Traditional hand-crafted no-reference metrics (e.g., Entropy, Mutual Information, VIF) and full-reference metrics (treating source pairs as pseudo-ground truths, e.g., SSIM, PSNR) often fail to reflect human perceptual preferences. They may even assign higher scores to visually inferior fused images.
2. Limitations of Current MLLM-based Approaches: Recent efforts to use Multimodal Large Language Models (MLLMs) for IVIF quality assessment typically rely on discrete one-hot supervision (predicting a single quality level) or scalar regression on aggregated scores.
   • Discrete Supervision: Forces fused images of similar quality into distinct, rigid rating levels (e.g., Level 4 vs. Level 3), collapsing subtle perceptual differences.
   • Scalar Regression: Learns from human ratings but fails to leverage the reasoning capabilities of MLLMs or encode the consensus (or ambiguity) of human judgments for specific images.

The core question addressed is: How can MLLMs produce continuous IVIF quality scores under supervision that encodes per-image annotation consensus?

2. Methodology: FuScore

The authors propose FuScore, an MLLM-based scorer that generates continuous quality scores rather than discrete levels. The framework consists of three main components:

A. Continuous Score Generation via Distributional Output

Instead of training the MLLM to emit a single quality token, FuScore utilizes the decoding probabilities over five predefined quality-level tokens ($L = \{1, 2, 3, 4, 5\}$).

• The model outputs a categorical distribution $q_i(l)$ over these tokens.
• The final continuous score $\hat{y}_i$ is computed as the expectation of this distribution: $\hat{y}_i = \sum l \cdot q_i(l)$.
• This allows the model to assign fine-grained, real-valued scores within the range $[1, 5]$, capturing subtle quality differences that discrete supervision would miss.

B. Sub-Dimension-Aware Soft Label Construction

To supervise the predicted distribution, FuScore constructs a "soft label" $p_i$ for each image. The center of this label is the overall human score, but its sharpness (width) is dynamically modulated by the agreement among four IVIF-specific sub-dimensions:

1. Thermal Retention
2. Texture Preservation
3. Artifacts
4. Sharpness

The authors define a consensus signal $\delta_i$ as the standard deviation of these four sub-dimension scores.

• High Consensus (Low $\delta_i$): When sub-dimensions agree, the soft label is sharply concentrated around the overall score.
• Low Consensus (High $\delta_i$): When sub-dimensions disagree, the label is broadened to reflect that the overall judgment is less consensual.
  This approach ensures the supervision target faithfully reflects the ambiguity inherent in human annotation for specific images.

C. Tripartite Training Objective

FuScore employs a composite loss function to address ranking at different scopes:

1. Per-Image Distributional Supervision ($\mathcal{L}_{KL}$): Aligns the predicted distribution with the soft Gaussian label using Kullback–Leibler divergence.
2. Within-Source-Pair Fidelity ($\mathcal{L}_{fid}$): Enforces Thurstone-style pairwise fidelity for images from the same source pair (same scene). This captures method-level ordering. The pairwise margin is derived from the soft-label widths, making the comparison uncertainty-aware.
3. Cross-Source-Pair Fidelity ($\mathcal{L}_{xfid}$): Enforces Thurstone-style fidelity across different source pairs (different scenes). This captures scene-level difficulty variations and ensures the model generalizes across scenes.

The total loss is a weighted sum: $\mathcal{L} = \mathcal{L}_{KL} + \lambda_{fid}\mathcal{L}_{fid} + \lambda_{xfid}\mathcal{L}_{xfid}$.

3. Key Contributions

1. FuScore: The first MLLM-based IVIF quality scorer trained with distribution-level supervision, capable of generating continuous scores for fused images.
2. Sub-Dimension-Aware Soft Labels: A novel label construction method that modulates label sharpness based on the agreement among four IVIF-specific sub-dimensions, allowing the supervision to reflect the consensus of human judgment.
3. Tripartite Training Objective: A joint optimization strategy combining per-image distributional matching, within-source-pair fidelity, and cross-source-pair fidelity, utilizing uncertainty-aware margins derived from sub-dimension agreement.

4. Experimental Results

Experiments were conducted on the EVAFusion benchmark (9,350 fused images, 850 source pairs) and a new SemanticRT 5-expert pilot dataset.

• State-of-the-Art Correlation: FuScore achieved the highest correlation with human visual preferences across all reported rank metrics (SRCC, PLCC, KRCC, PairAcc, Per-SP $\tau$).
  • On EVAFusion, the 3-seed ensemble achieved an SRCC of 0.648, outperforming the strongest external baseline (ViT-L+MLP) by +0.094.
  • It significantly outperformed zero-shot natural-image IQA models (e.g., Q-Align, DeQA-Score) and hand-crafted metrics, indicating that natural-image pretraining does not transfer well to IVIF without in-domain supervision.
• Generalization:
  • Cross-Method: In a leave-K-method-out evaluation, FuScore maintained high correlation (SRCC 0.628), only 0.020 below its in-distribution performance.
  • Cross-Dataset: On the SemanticRT pilot (unseen data), FuScore achieved an SRCC of 0.561, outperforming the ViT-L+MLP baseline (0.471) by +0.090.
• Ablation Studies:
  • The tripartite objective (specifically the cross-source-pair fidelity) was shown to be the primary driver for ranking gains.
  • The sub-dimension-aware soft labels successfully modeled annotation ambiguity. The authors demonstrated that the predicted uncertainty ($\hat{\sigma}$) correlates positively with human inter-rater disagreement on the SemanticRT pilot ($\rho = 0.42$), confirming that high-dimensional conflict ($\delta_i$) reflects genuine human ambiguity rather than just multi-attribute trade-offs.
  • A "data-side floor" was observed: on images with high sub-dimension disagreement (high-$\delta$), all 11 evaluated methods (including FuScore) plateaued at a low SRCC band, suggesting a limit imposed by the inherent inconsistency of human annotations for these specific images.

5. Significance and Claims

The paper claims that FuScore represents a significant step forward in IVIF quality assessment by:

• Bridging the Human-Alignment Gap: Moving beyond rigid discrete levels and scalar regression to capture the nuance of human perception and the consensus of expert judgment.
• Leveraging MLLM Reasoning: Utilizing the linguistic and visual reasoning capabilities of MLLMs (specifically Qwen3-VL) in a way that is specifically tailored to the IVIF domain through sub-dimension-aware supervision.
• Addressing Annotation Ambiguity: Explicitly modeling the uncertainty in human ratings, which allows the model to learn more robustly from noisy or conflicting expert data.

The authors note that while FuScore achieves state-of-the-art results, the performance ceiling on high-ambiguity images suggests that future improvements may depend on richer multi-rater annotations that can disentangle aggregation artifacts from inherent human ambiguity. The paper also acknowledges the inference cost of using large MLLMs, suggesting that real-time deployment would require smaller backbones.