💻 computer science

Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimization

This paper proposes DPOFusion, a direct preference optimization framework that integrates property-aligned and preference-controllable latent diffusion models to achieve adaptive infrared and visible image fusion capable of aligning with heterogeneous human and machine vision demands.

Original authors: Weijian Su, Songqian Zhang, Yuqi Han, Jian Zhuang, Yongdong Huang, Qiang Zhang

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weijian Su, Songqian Zhang, Yuqi Han, Jian Zhuang, Yongdong Huang, Qiang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two cameras taking a picture of the same scene at the same time. One is a visible camera (like your phone), which sees great colors and textures but struggles in the dark. The other is an infrared camera, which sees heat signatures clearly in the dark but looks like a blurry, black-and-white thermal ghost.

Image Fusion is the art of blending these two photos into one perfect image that has the best of both worlds.

The Problem: "One Size Does Not Fit All"

The paper points out a major headache with current fusion methods: they are rigid.

Imagine you ask a chef to cook a steak.

A human might say, "I want it medium-rare with a nice sear."
A security guard might say, "I don't care about the taste; I just need to see the person's face clearly to identify them."
A self-driving car might say, "I need to see the edge of the road perfectly to avoid hitting a pedestrian."

Current fusion models are like a chef who can only cook one specific way. If you want a different result, you have to hire a completely new chef (train a new AI model) with a new recipe. This is slow, expensive, and inflexible.

The Solution: "Fusion in Your Way"

The authors propose a new system called DPOFusion. Think of this as a super-chef with a "taste-bud" adjustment dial.

Instead of cooking a new meal from scratch for every customer, this chef starts with a high-quality base dish and then instantly tweaks it based on exactly what you ask for.

Here is how it works, broken down into simple steps:

1. The "Tasting Menu" Generator (PALDM)

First, the system creates a "tasting menu" of many different fusion results.

It takes the two source photos (visible and infrared).
It uses a special AI (a Latent Diffusion Model) to generate a wide variety of blended images. Some might be 90% infrared, some 50/50, some with extra texture, some with better contrast.
Analogy: Imagine a painter who quickly sketches 50 different versions of a landscape, each with slightly different lighting or focus. This gives the system a huge pool of options to choose from.

2. The "Taste Test" (Collecting Preferences)

Now, the system needs to know which version is "best." But "best" depends on who is asking.

For Humans: A person looks at the sketches and says, "I like the texture on the car here, but the sky looks weird." They mark the good parts and the bad parts.
For Machines: A security camera (Object Detection AI) looks at the sketches and says, "I can't see the person in this one, but I can see them clearly in that one."
For AI Critics: A smart language model (VLM) acts like a food critic, ranking the images based on technical quality (sharpness, no noise, natural colors).

3. The "Magic Dial" (Direct Preference Optimization - IDPO)

This is the paper's secret sauce. Instead of retraining the whole chef, they use a technique called Direct Preference Optimization (DPO).

The Old Way: To teach a model a new preference, you usually have to retrain it from the ground up, which is like sending the chef back to culinary school for a year.
The DPO Way: The system looks at the "Winning" image (what you liked) and the "Losing" image (what you didn't like). It then makes a tiny, precise adjustment to the chef's brain to make the next image look more like the winner.

The "Mask" Trick (Instance Direct Preference Optimization):
Sometimes you only want to change part of the image.

Example: "Make the car clearer, but don't touch the sky."
The system uses a mask (like a stencil). It tells the AI: "Only adjust the pixels inside this stencil to match the preference. Keep everything outside the stencil exactly the same."
This ensures that if you ask for better car details, you don't accidentally ruin the sky or the background.

The Results

The paper tested this "Magic Dial" system on real-world datasets (like night-time street scenes).

Human Feedback: When asked to make images look more natural, the system adjusted the colors and textures perfectly.
Machine Feedback: When asked to help a self-driving car "see" better, the system tweaked the image to highlight pedestrians and cars, making the car's detection software much more accurate.
Versatility: The same base model could switch between "Human Mode," "Security Mode," and "Driving Mode" just by changing the prompt, without needing to be retrained.

Summary

DPOFusion is a flexible image blender that learns from feedback. Whether you are a human wanting a pretty photo, a security system needing clear details, or a robot needing to avoid obstacles, this system can instantly adjust the fusion result to fit your specific needs, all while keeping the rest of the image perfect. It moves us from "one model for everyone" to "fusion in your way."

Technical Summary: DPOFusion

Problem Statement

Infrared and visible image fusion (IVIF) aims to integrate complementary spectral information to enhance human perception and improve downstream vision tasks. However, existing methods face a critical limitation: they struggle to flexibly accommodate heterogeneous demands. Different users and applications prioritize conflicting objectives, such as overall image quality, natural appearance, or specific suitability for machine vision tasks like object detection and segmentation. Current learning-based approaches typically require separate models and complex training strategies for each specific demand, lacking a unified framework capable of adaptive fusion that aligns with diverse human and machine preferences.

Methodology

The authors propose DPOFusion, a framework that leverages Direct Preference Optimization (DPO) to align image fusion with heterogeneous demands. The framework consists of two primary components and a data collection pipeline:

1. Property-Aligned Latent Diffusion Model (PALDM)

PALDM serves as the candidate generator. It is designed to produce diverse, high-quality fusion results with varying properties.

Prior Latent Fusion: A prior latent fusion model ( $\epsilon_{lfm}$ ) is first trained using a joint fusion loss to guide the generation process.
Joint Conditional Loss: To generate diverse candidates, PALDM employs a joint conditional loss. It trains a latent diffusion model to denoise both standard fused latents and property-interpolated latents (created by linearly interpolating source latents and the prior fused latent). This allows the model to align generated outputs with specific property-descriptive text prompts (e.g., "mostly infrared information").

2. Preference-Controlled Latent Diffusion Model (PCLDM)

PCLDM is a fine-tuning module that adjusts the fusion results based on specific preference signals.

Architecture: It initializes from the frozen PALDM ( $\epsilon_{ref}$ ) and introduces a trainable branch connected via a zero-initialized 1 $\times$ 1 convolution. This structure preserves the foundational fusion capabilities while allowing rapid adaptation to new preferences.
Instance Direct Preference Optimization (IDPO): This is the core innovation. Unlike standard DPO which optimizes globally, IDPO applies preference supervision only within a masked region of interest while enforcing strict pixel-level consistency elsewhere.
- Loss Function: The IDPO loss ( $L_{IDPO}$ ) combines a preference alignment loss for the masked region (encouraging the model to prefer the "winning" sample over the "rejected" one) and a consistency loss for the unmasked regions (ensuring the output remains close to the frozen reference model). This balances localized adaptation with global fidelity.

3. Preference Data Collection

The framework collects heterogeneous preference data from three sources to create training pairs ( $I_w^0 \succ I_l^0$ ) with corresponding masks ( $I_m$ ):

Human Feedback (RLHF): Human annotators select preferred regions in candidate images.
Vision-Language Models (RLVF): VLMs (specifically QWEN3-Omni-Think) rank candidates based on technical image quality (sharpness, artifacts, etc.).
Task-Driven Models (RLDF): Downstream models (YOLOv11 for detection, SegFormer for segmentation) evaluate candidates. Preferred/rejected samples are determined by task performance metrics (mAP, mIoU), and masks are derived from target regions or detection patches.

Key Contributions

Unified Framework: DPOFusion presents a single framework capable of adapting to diverse user preferences (human, VLM) and machine vision tasks (detection, segmentation) without requiring separate models for each.
Instance Direct Preference Optimization (IDPO): The authors introduce IDPO, a novel adaptation of DPO for image fusion. By restricting preference optimization to specific masked regions, IDPO achieves precise local alignment without degrading global image structure or coherence.
Property-Aligned Generation: The integration of PALDM with joint conditional loss enables the generation of a rich pool of diverse candidates, facilitating effective preference annotation and fine-tuning.
Task-Oriented Adaptability: The framework demonstrates the ability to fine-tune fusion outputs specifically to enhance downstream task performance (e.g., improving segmentation of small targets or detection of dense objects) while maintaining visual quality.

Experimental Results

The authors evaluated DPOFusion on real-world datasets (LLVIP, MSRS, RoadScene, M3FD) against nine state-of-the-art methods (e.g., U2Fusion, DDFM, SHIP, Text-IF).

Human and VLM Alignment: Quantitative metrics (Entropy, Standard Deviation, Average Gradient, MUSIQ, CNNIQA) show that DPOFusion variants (RLHF and RLVF) achieve state-of-the-art performance. Qualitative results indicate superior texture preservation and natural appearance, particularly in low-light and complex illumination scenarios.
Downstream Task Performance:
- Semantic Segmentation: The RLDF-Seg variant achieved a 0.5% improvement in mIoU compared to baselines, with qualitative results showing better segmentation of small targets.
- Object Detection: The RLDF-OD variant achieved a 4.2% improvement in mAP, demonstrating effective feature capture for dense and small targets.
Ablation Studies: Experiments confirmed that the joint conditional loss in PALDM yields higher-quality candidates and that IDPO significantly outperforms standard DPO and contrastive loss in aligning with preferences while preserving global consistency.
Efficiency: While diffusion-based, DPOFusion operates in a compressed latent space, reducing inference time to 1.7s per image (compared to 7.8s for DDFM), making it more efficient than other generative fusion methods.

Significance and Claims

The paper claims that DPOFusion successfully addresses the "open and challenging problem" of achieving adaptive fusion that aligns with various preferences from both human and machine vision. By leveraging DPO, the framework eliminates the need for explicit reward models and complex joint-optimization pipelines with downstream tasks.

The authors position DPOFusion as a unified and flexible solution for "fusion in your way," capable of fine-tuning a foundational model to satisfy specific requirements—whether driven by human visual perception, language instructions, or downstream task metrics—through minimal fine-tuning. The work sets a new benchmark for adaptive fusion quality and task-oriented transferability in the field of infrared and visible image fusion.