Technical Summary: HP-UniIF

Problem Statement

General image fusion aims to synthesize complementary information from multiple source images into a single, informative output. However, real-world deployment scenarios often present a "M3 setting" where a single system must simultaneously handle:

1. Heterogeneous Fusion Tasks: Such as infrared-visible fusion (VIF), multi-exposure fusion (MEF), and multi-focus fusion (MFF).
2. Diverse Degradations: Inputs may suffer from noise, blur, low illumination, haze, or excessive illumination.
3. Downstream Applications: The fused output must often serve specific perception tasks like object detection or semantic segmentation.

Existing unified frameworks struggle to accommodate these orthogonal objectives (fusion quality, restoration robustness, and task-specific adaptation) within a single pipeline. They often result in entangled representations where optimizing for one objective degrades performance in others, or they rely on specialized models for individual settings rather than a unified solution.

Methodology: HP-UniIF

The authors propose HP-UniIF, a unified vision framework built upon pretrained latent diffusion models. It leverages diffusion priors to bridge heterogeneous fusion, visual restoration, and downstream perception. The core innovation is a depth-wise hierarchical conditional modulation strategy that decouples task, degradation, and application objectives across different network stages using three distinct prompt-learning modules:

1. Framework Overview

HP-UniIF utilizes a Variational Autoencoder (VAE) and a denoising U-Net. Two source images are encoded into latent codes, which are fused to initialize the diffusion process. The U-Net then performs refinement steps to generate the final fused latent representation, which is decoded back to the image space.

2. Hierarchical Prompt Learning Modules

The framework introduces three specialized components organized by network depth:

• Task Prompt Modulation (TPM) - Bottleneck/Decoder Stages:

  • Function: Adapts the shared generative pathway to specific fusion tasks (e.g., switching between VIF and MEF).
  • Mechanism: Selects an initial task prompt from a learnable bank and progressively refines it based on the current feature state (from coarse semantics to fine details). It predicts channel-wise affine parameters and an injection gate to modulate features, ensuring the backbone adapts to the specific information-selection strategy required by the task without excessive perturbation.

• Degradation Prompt Router (DPR) - Shallow Layers:

  • Function: Addresses input degradations (noise, blur, etc.) before cross-source fusion occurs.
  • Mechanism: Constructs a degradation descriptor from source features and their discrepancies. It retrieves the most relevant degradation prompts from a level-specific pool using a query-router mechanism. These prompts guide a lightweight restoration block to predict source-specific corrections and injection gates, restoring reliable evidence before fusion.

• Application Prompt Bank (APB) - Decoding Stages:

  • Function: Aligns the generation process with specific downstream tasks (e.g., object detection, semantic segmentation).
  • Mechanism: Injects application-specific semantic guidance into the decoder features. It employs a dual-branch structure: a prompt-injection branch to introduce task priors and a content-retention branch to preserve the fused visual evidence. This allows the model to adapt to downstream requirements without altering the core fusion or restoration pathways.

3. Training Strategy

The model is trained in three decoupled stages to prevent gradient interference:

1. S1 (Multi-Task Fusion): Jointly trains the fusion layers and TPM on clean data.
2. S2 (Degradation Restoration): Freezes the fusion path and optimizes DPR on paired clean/degraded data.
3. S3 (Application Adaptation): Freezes fusion and restoration paths, optimizing only APB and application-specific components using downstream task objectives (e.g., detection loss). New applications can be added by learning new prompt entries with minimal trainable parameters.

Key Contributions

1. Unified Framework: HP-UniIF is a unified framework that jointly supports multiple fusion tasks, robust restoration under complex degradations, and adaptation to multiple downstream applications using diffusion priors, addressing the M3 challenge.
2. Hierarchical Prompt Learning: The introduction of TPM, DPR, and APB allows for structurally separated adaptation paths. This design mitigates interference between fusion, restoration, and application objectives while maintaining a strong shared visual prior.
3. Modular Extensibility: The architecture allows for the addition of new downstream applications or degradation types by simply learning new prompt entries and lightweight adapters, keeping the shared backbone frozen.

Experimental Results

Extensive experiments were conducted on three fusion tasks (VIF, MEF, MFF) under both clean and degraded conditions, with downstream evaluations on object detection (YOLOv5) and semantic segmentation (DeepLabV3+).

• Visual Quality: Qualitative comparisons show HP-UniIF produces visually faithful results, preserving salient targets (e.g., pedestrians in infrared), recovering details in underexposed regions, and maintaining sharp focus in multi-focus scenarios. It effectively suppresses noise and artifacts compared to state-of-the-art methods.
• Quantitative Performance: Under degraded conditions, HP-UniIF achieved the best performance across all 17 evaluation metrics (including EN, QAB/F, MI, VIFF, PSNR, and SSIM) across the three fusion tasks.
• Downstream Performance: In downstream tasks, HP-UniIF achieved the highest mAP and mIoU scores. Notably, it showed significant improvements in mAP@.5:.95, indicating better object localization, and achieved the best class-wise IoU in 5 out of 8 semantic segmentation categories.
• Ablation Studies: Experiments confirmed that the diffusion prior provides a strong structural foundation. TPM and DPR were shown to be complementary, with their joint use yielding the strongest fusion performance. APB was found to improve downstream task performance without significantly disrupting intrinsic fusion quality.

Significance and Claims

The paper claims that HP-UniIF addresses a critical gap in current image fusion research: the lack of a unified system capable of handling the "M3 setting" (Multi-task, Multi-degradation, Multi-application) effectively. While prior works like CCF and DRMF have explored conditional diffusion or joint restoration and fusion, HP-UniIF distinguishes itself by integrating task, degradation, and application prompts within a single shared diffusion prior through a hierarchical design. By leveraging diffusion priors and a hierarchical prompt-learning strategy, the authors demonstrate that it is possible to decouple orthogonal objectives that typically interfere with one another.

The significance lies in the framework's ability to produce visually faithful results while preserving task-relevant semantics, offering a flexible solution for real-world deployment where inputs are often degraded and the end-use case varies. The authors position HP-UniIF not just as a performance booster, but as a structural paradigm shift towards unified, adaptable vision systems that can support diverse fusion tasks and downstream applications simultaneously.