💻 computer science

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

The paper introduces P2Fusion, a novel prompt-based framework that leverages dual intrinsic priors and a Teach-to-Fuse mechanism with gated dynamic expert recalibration to achieve state-of-the-art infrared-visible image fusion performance and significantly enhance downstream object detection robustness.

Original authors: Yi Shi, Huichao Xie, Yuqing Wang, Mingyu Wang, Kaihui Yang, Yu Liu, Ruitao Lu, Lizhe Li, Junwei Han, Dingwen Zhang

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi Shi, Huichao Xie, Yuqing Wang, Mingyu Wang, Kaihui Yang, Yu Liu, Ruitao Lu, Lizhe Li, Junwei Han, Dingwen Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take the perfect photo of a spooky, foggy night. You have two cameras: one that sees in the dark like a bat (infrared), which is great at spotting warm things like people or cars, but looks blurry and lacks detail. The other is a regular camera (visible light), which sees sharp textures like tree branches and road signs, but gets completely blinded by the darkness or smoke. Image fusion is the art of magically combining these two photos into one super-image that has the best of both worlds. For years, scientists have tried to build computers that can do this automatically. The goal is to help robots, self-driving cars, and drones "see" clearly in bad weather, at night, or through smoke, so they don't crash or miss important targets. However, teaching a computer to blend these two very different types of vision without messing up the details has been a tricky puzzle.

The researchers behind this paper, called P²Fusion, decided to stop forcing the computer to follow rigid, pre-written rules. Instead, they invented a new way to teach the AI how to blend these images by using "dynamic prompts," which are like helpful, shifting hints rather than strict orders. They call their method "Teach-to-Fuse."

Here is the problem they are solving: In the past, scientists tried to guide the computer by giving it static "prior knowledge"—like a fixed map of where objects should be or a rigid rule saying "always keep the edges sharp." The authors argue that this is like trying to drive a car with a map that never updates; if the road changes, the driver gets confused. Other methods tried to use massive, pre-trained AI models (like the ones that recognize cats or dogs) to give hints, but the authors found these hints were often too vague and high-level, missing the tiny, pixel-level details needed for a perfect photo.

To fix this, P²Fusion uses a clever two-step strategy. First, it acts like a smart teacher that doesn't just hand out answers but teaches the AI how to learn from the images themselves. It uses two "teachers":

The Thermal Teacher: This teacher looks at the infrared image and highlights the "hot" spots (like a person or a car) that need to be preserved.
The Quality Teacher: This teacher looks at the visible image and points out which parts are clear and sharp, and which parts are blurry or dark.

Instead of hard-coding these hints, the system turns them into "prompts"—flexible, learnable signals that guide the AI as it works. Think of it like a conductor leading an orchestra. Instead of telling every musician exactly what note to play at every second (which would be rigid and prone to error), the conductor gives dynamic cues based on how the music is actually sounding, helping the musicians adjust in real-time.

The second part of their invention is a special module called GDER (Gated Dynamic Expert Recalibration). Imagine a team of two specialists working on a puzzle. One specialist is an expert at finding the "hot" targets (the Modality Expert), and the other is an expert at seeing the overall structure and texture (the Attention Expert). In many old systems, these two would just mash their ideas together, often causing confusion. P²Fusion uses a "gating" mechanism—a smart referee—that decides how much weight to give each specialist's opinion at every single moment. If the scene is smoky, the referee might listen more to the thermal expert. If the scene is bright but cluttered, it might listen more to the texture expert. This allows the system to fix its own mistakes and balance the two types of vision perfectly.

The authors tested their new system on five different datasets, including scenes with heavy smoke, extreme brightness, and night-time driving. They found that P²Fusion consistently produced better results than the current best methods. In fact, it topped the charts in 14 out of 20 key measurement categories across these tests. It didn't just look better; it actually helped computers detect objects more accurately. For example, when used to help a drone spot vehicles, it improved detection accuracy by 3.2% on one dataset and 0.9% on another. Even when tested on completely new, unseen environments (like aerial drone footage it had never seen before), the system remained robust and didn't crash or fail.

In short, the paper suggests that by moving away from rigid, static rules and instead using flexible, image-based "prompts" guided by a smart, self-correcting team of experts, we can create much better tools for seeing in the dark and through the fog. The authors believe this approach solves the conflict between keeping the image sharp and keeping the important targets visible, offering a more adaptable solution for the future of autonomous machines.

Technical Summary of P²Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

Problem Statement

Infrared-Visible Image Fusion (IVIF) aims to integrate complementary thermal radiation and textural details into a comprehensive representation. However, existing prior-guided methods face three fundamental limitations:

Static Constraints: Methods relying on explicit modality-specific hard-constraints (e.g., segmentation masks) often treat priors as rigid regularization terms. This forces the network to prioritize fitting prior distributions over pixel-level feature integration, leading to the loss of fine-grained textures.
Optimization Conflicts: Joint optimization with downstream tasks (e.g., object detection) frequently introduces multi-objective conflicts where the drive for task-level accuracy compromises background detail preservation.
Granularity Mismatch: Recent trends utilizing extrinsic semantic guidance from large-scale foundation models (e.g., CLIP, DINO) introduce high-level abstract representations that deviate from the pixel-level essence of image fusion, resulting in a mismatch between semantics and low-level textural reconstruction.

Methodology

The authors propose P²Fusion, a framework grounded in the "Teach-to-Fuse" (T2F) paradigm. Instead of imposing static penalties, the method reformulates IVIF by distilling intrinsic, dynamic prompts from the input images themselves.

1. Dual-Teacher Prompt Distillation
The framework replaces rigid priors with learnable, process-oriented guidance signals derived from two complementary intrinsic attributes:

Infrared Saliency Prior ( $T_{ir}$ ): A "Thermal Saliency Teacher" (IST), based on a SegFormer-B2 model, generates saliency maps to localize high-thermal-radiation targets (pedestrians, vehicles). These maps are post-processed to treat specific targets as foreground and serve as spatial prompts.
Visible Quality Prior ( $T_{vis}$ ): A "Visible-Quality Teacher" (VQT), utilizing a BRISQUE-based evaluator, generates a global quality score to assess textural reliability (e.g., identifying under/over-exposed areas).
These priors are projected into learnable prompts ( $P_{ir}, P_{vis}$ ) and embedded into the feature streams, transforming static constraints into dynamic regulators.

2. Adaptive Fusion with Gated Dynamic Expert Recalibration (GDER)
To handle the heterogeneity between thermal contours and textural details, the architecture employs a Gated Dynamic Expert Recalibration (GDER) module. Unlike traditional Mixture-of-Experts (MoE) used for capacity expansion, GDER functions as a self-correcting feedback mechanism for feature decoupling and refinement:

Prior-Responsive Modality Expert ( $E_{mod}$ ): Explicitly consumes dynamic prompts to reinforce modality-specific salient information (e.g., thermal targets or high-contrast textures).
Prior-Agnostic Attention Expert ( $E_{att}$ ): Operates independently of prompts, utilizing a global attention mechanism (CBAM) to mine intrinsic long-range dependencies and fine-grained structural details, acting as a contextual compensator.
Gating Mechanism: A gating network computes dynamic weights based on the mutual agreement between local features and global prompts. This allows the network to adaptively recalibrate the fusion strategy, decoupling conflicting features and rectifying biased signals.

3. Optimization Objectives
The total loss function ( $L_{total}$ ) combines perceptual constraints (Gradient, Intensity, and SSIM losses) with Prompt Distillation Losses. The distillation losses (MSE for visible quality, Binary Cross-Entropy for infrared saliency) ensure the learnable prompts align with the teacher signals without imposing hard-coded penalties.

Key Contributions

Teach-to-Fuse Paradigm: The authors propose a shift from static prior constraints to dynamic, prompt-based guidance. This approach transforms penalties into process-oriented guidance, maximizing the mining of intrinsic modality attributes.
GDER Module: The development of a specialized MoE-based self-correction mechanism that synergizes a prior-guided Modality Specialist with a global-aware Attention Specialist, enabling adaptive decoupling of conflicting features.
Comprehensive Benchmarking: The framework is evaluated across five public datasets (MSRS, M3FD, FMB, RoadScene, DroneVehicle) and over ten benchmark tasks, demonstrating robustness in diverse scenarios including smoke occlusion and extreme overexposure.

Experimental Results

Fusion Quality: P²Fusion achieves state-of-the-art (SOTA) performance on 12 out of 16 key evaluation metrics across the four main datasets (MSRS, M3FD, FMB, RoadScene) and 2 out of 4 metrics on the DroneVehicle dataset, totaling 14 out of 20 key metrics. It consistently outperforms 12 existing SOTA methods (including SAGE, DiTFuse, and FreeFusion) in visual fidelity, preserving both infrared salient targets and visible textures.
Downstream Perception: The fused images significantly enhance downstream tasks:
- Object Detection: +3.2% mAP on MSRS, +0.5% mAP on M3FD, and +0.9% mAP on DroneVehicle.
- Semantic Segmentation: Consistent gains in mPA and mIoU on MSRS and FMB datasets.
Robustness & OOD: In Out-of-Distribution (OOD) verification on the DroneVehicle dataset (featuring aerial perspectives and severe visibility degradation), P²Fusion maintained SOTA performance without domain-specific fine-tuning, demonstrating superior adaptability in unseen, hostile environments.

Significance and Claims

The paper claims that P²Fusion effectively addresses the inherent prior-dependency of traditional frameworks by shifting the paradigm from "static constraints" to "dynamic intrinsic prompting." By returning to the essence of image data and utilizing endogenous, dynamic regulators, the method achieves a robust balance between visual aesthetics and task performance. The authors assert that this approach bridges the gap between high-level physical guidance and adaptive feature representation, providing a versatile solution for multi-modal perception that mitigates optimization conflicts and granularity mismatches found in prior-guided methods.