A Dual-Stream Network with Spatial Alignment and Swin Transformer Fusion for Visible-Infrared Object Detection
This paper proposes a dual-stream visible-infrared object detection network that integrates wavelet-based feature modeling, an OffsetNet module for spatial alignment, and a SwinFusion module for cross-modal feature fusion to effectively address spatial misalignment and modality discrepancies, thereby improving detection accuracy.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of machine vision, cameras are often specialized tools, each with a specific way of seeing the world. Standard cameras, like the ones in our phones, capture visible light, recording the rich colors, textures, and fine details of a scene. However, these images can vanish in the dark, blur through heavy fog, or become washed out by harsh glare. Infrared cameras offer a different perspective. Instead of seeing reflected light, they detect heat radiation, allowing them to "see" warm objects like people or engines even in total darkness or through smoke. While each type of camera has its strengths, relying on just one leaves gaps in our understanding of a scene. For decades, researchers have tried to combine these two views into a single, more reliable picture, a process known as visible-infrared object detection. The goal is to create systems that can spot cars, pedestrians, or obstacles in any weather, day or night, by merging the color and texture of visible light with the thermal clarity of infrared heat.
The challenge, however, is that these two cameras rarely see the world in perfect sync. Because they are often mounted on different sensors or have different lenses, the images they capture can be slightly shifted, tilted, or scaled differently. Imagine trying to overlay two maps of the same city where one is slightly larger and shifted to the left; the streets would not line up, and the landmarks would be in the wrong places. When a computer tries to merge these misaligned images, the result is a confused mess where the edges of objects blur and the system fails to recognize what it is looking at. Furthermore, the two types of images look fundamentally different to a computer: one is full of color and texture, while the other is a map of heat. Simply stacking these images on top of each other does not work well because the computer struggles to understand how the warm shape of a person in the infrared image corresponds to the detailed clothing of that same person in the visible image.
To solve this, a team of researchers from Soochow University and other institutions has developed a new method that acts like a skilled editor, carefully aligning and blending these two distinct views before presenting them to the detection system. Their approach, detailed in a recent study, moves away from simple mixing and instead treats the alignment and fusion as a multi-step process. First, the system breaks down the images into different layers of detail, separating the broad shapes from the fine edges. This allows the computer to preserve the structural integrity of objects while it works on the harder task of matching them up. The researchers then introduced a specific component, which they call a spatial alignment module, to handle the shifting problem. Instead of trying to physically move the entire image, this module predicts tiny, task-specific adjustments for every part of the visible image, effectively sliding the details into the correct position relative to the infrared image. It is a bit like adjusting the focus on a camera lens, but done digitally and instantly for every single part of the picture to ensure the edges of a car or a person line up perfectly with their thermal signature.
Once the images are aligned, the system uses a powerful fusion engine to combine them. This engine is designed to first understand the context within each image type on its own—learning the patterns of heat in the infrared view and the patterns of texture in the visible view—before it attempts to merge them. By understanding each perspective individually first, the system can then blend them in a way that highlights the most useful information from both. For instance, it might use the sharp edges from the visible image to define the shape of an object while using the heat signature from the infrared image to confirm that the object is actually a living being or a vehicle. This step-by-step approach ensures that the final combined image is not just a sum of two parts, but a coherent, enhanced view that retains the best qualities of both.
The researchers tested this new system on four different public datasets, which included real-world scenarios ranging from autonomous driving on city streets to aerial surveillance from drones. These datasets contained thousands of image pairs taken in various conditions, from bright daylight to pitch-black nights, and included objects of many different sizes. The results showed that their method consistently outperformed existing techniques. On a dataset focused on autonomous driving, the new system achieved a detection accuracy score of 80.9 percent, which was higher than previous methods. In low-light pedestrian detection, it matched the performance of the best existing models, and in aerial vehicle detection, it improved the ability to spot small objects from high above. The study also included detailed tests to prove that each part of their system was necessary. When they removed the alignment module, the system struggled with misaligned images. When they removed the advanced fusion engine, the system missed subtle details. The combination of both components was what drove the improvement.
While the system is highly effective, the researchers acknowledge that it comes with a cost. The complex process of aligning and fusing the images requires more computing power and time than simpler methods, which could make it harder to run on small, battery-powered devices in real-time. Additionally, the alignment module is designed to fix small, local shifts rather than massive, global misalignments, meaning it works best when the cameras are already roughly pointed in the same direction. Despite these limitations, the work demonstrates a clear path forward for machine vision. By treating the alignment and fusion of different camera types as a careful, structured process rather than a simple mix, the researchers have shown that computers can learn to see the world more clearly, combining the color of the day with the heat of the night to create a safer, more reliable view for everything from self-driving cars to security drones.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.