Current Injection Spiking Neural Network for Infrared and Visible Image Fusion

Imagine you are trying to build the ultimate camera for a robot that needs to see perfectly in any situation. You have two different eyes: one is a "visible" eye that sees amazing colors and textures, like a human does, but it goes blind in the dark or through smoke. The other is an "infrared" eye that sees heat, so it can spot a person in total darkness or through fog, but it looks like a blurry, gray thermal blob with no fine details. The goal of Image Fusion is to mash these two eyes together into one super-vision system that has the best of both worlds.

For a long time, scientists have been trying to do this using Artificial Neural Networks (ANNs). Think of these like a super-fast, super-tired accountant who adds up every single number in a spreadsheet, no matter how small or unimportant. It works well, but it uses a lot of energy, like running a marathon just to check the mail.

Enter Spiking Neural Networks (SNNs). These are inspired by how our actual brains work. Instead of constantly adding numbers, they act like a room full of people waiting to shout. They only "fire" a signal (a spike) when something interesting happens. This makes them incredibly energy-efficient, like a lightbulb that only turns on when someone walks into the room. However, there's a catch: because they only shout when they are loud enough, they might miss the quiet whispers of important information. If a thermal signal is too weak to make the neuron shout, it gets ignored, and the robot loses that crucial detail.

This is where a new paper comes in to solve that problem. The researchers, led by Rui Zhao and colleagues, propose a clever trick called CIS-Fuse. Instead of waiting for the neuron to shout (fire a spike) before mixing the two types of vision, they mix the signals before the shout happens. They use a mechanism called Current Injection, where the quiet, weak signal from one eye is gently injected into the "battery" (membrane potential) of the neuron in the other eye. This way, even if the signal is too quiet to shout on its own, it can still help push the neuron over the edge to fire when combined with the other eye's signal.

The result is a system that keeps the super-low energy savings of the "shouting" brain but doesn't lose the fine details. The team tested this on four different datasets and found that their method produces fused images just as good as the heavy, energy-hungry accountants (ANNs), but with roughly 10 times less energy to run. They also showed that robots using these fused images get better at spotting cars and people in the dark, proving that you don't have to burn a lot of power to see clearly.

The Problem: The "Too Quiet to Shout" Dilemma

To understand why this paper is a big deal, we have to look at the tension between two very different ways of processing information.

On one side, you have the Artificial Neural Networks (ANNs) that power most modern AI. These are like a massive, continuous stream of water. Every drop of water (every piece of data) is measured and mixed together. This is great for capturing every tiny detail, whether it's a bright light or a faint shadow. But it's expensive. It requires a lot of computing power, which drains batteries quickly. This is a problem for things like drones or self-driving cars that need to run all day on limited power.

On the other side, you have Spiking Neural Networks (SNNs). These are like a digital version of a nervous system. Neurons in an SNN sit quietly until they receive enough input to cross a "threshold." Once they cross that line, they fire a single, binary spike (a 1). If they don't cross the line, they stay silent (a 0). This is incredibly efficient because the system only does work when something interesting happens.

The problem arises when you try to use SNNs for Image Fusion. Fusion requires taking a weak signal from the infrared camera (maybe a faint heat signature of a person) and a strong signal from the visible camera (the texture of a tree) and blending them perfectly. In a standard SNN, if the infrared signal is too weak to make the neuron fire on its own, it gets discarded. It's like trying to mix a whisper with a shout; if you only listen to the shout, you miss the whisper. The paper argues that this "binary" nature of SNNs throws away the very complementary information that makes fusion useful.

The Solution: Mixing Before the Shout

The authors of this paper, CIS-Fuse, realized that the solution lies in the membrane potential. In a biological neuron (and in SNNs), before a neuron fires, it accumulates charge in its "battery" (the membrane potential). This battery holds onto the input even if it's not strong enough to trigger a shout yet.

The paper proposes a new operator called Current Injection Spiking (CIS). Here is how it works in simple terms:

1. The Setup: Imagine two streams of data: one from the Infrared camera and one from the Visible camera.
2. The Injection: Instead of letting them fight it out to see who gets to fire, the system takes the "auxiliary" stream (say, the infrared) and injects it as a "gated current" into the "battery" of the main stream (the visible).
3. The Gate: There is a smart gatekeeper (a learned gate) that decides how much of the auxiliary signal to let in. It also has a special dial (a learnable scale factor) for every single channel of information, allowing the system to say, "Hey, this specific detail is important, let's boost it," or "This one isn't needed, keep it low."
4. The Fusion: The two signals mix inside the battery before the neuron decides to fire. This means a weak infrared signal can help push a neuron over the edge to fire, even if the visible signal alone wasn't quite enough.

This approach preserves the "fine-grained" responses that would otherwise be lost. It's like having a team of people where a quiet person can still contribute to the final decision if they are standing next to someone loud, rather than being ignored because they didn't speak up first.

The Architecture: A Two-Lane Highway with Specialized Drivers

To make this work effectively, the researchers built a specific network architecture. They didn't just throw the idea into a random mix; they designed a dual-branch system with a clever twist.

They created two parallel paths (branches) that process the images:

• The Shallow Branch: This path has fewer layers (just one block of their special fusion module).
• The Deep Branch: This path is longer, stacking three blocks of the fusion module on top of each other.

At first, you might think, "Why have two different lengths?" The paper suggests that this asymmetry allows the network to naturally specialize. After training, the "Shallow" branch and the "Deep" branch started behaving differently. The shallow branch learned to do a mild mix, while the deep branch learned to do a heavy, intense mix. It's like having two chefs in a kitchen: one is a quick, light-handed mixer, and the other is a heavy-duty blender. Together, they cover all the bases.

The paper also introduces a Reparameterized Output Head. This is a technical trick that makes the system smarter during training but simpler during use. During training, the system uses a complex, multi-branch structure to learn the best way to combine the images. But once training is done, the system "collapses" all those branches into a single, simple convolution layer. This means the final product is just as fast and simple as a standard layer, without the extra baggage of the training complexity.

The Results: High Quality, Low Energy

The researchers tested CIS-Fuse on four different benchmarks (datasets of infrared and visible images) and compared it against 15 other state-of-the-art methods, including heavy hitters like Text-IF, DCEvo, and S4Fusion.

Visual Quality:
The results showed that CIS-Fuse produces fused images that are just as good as the best ANN-based methods. In fact, it ranked at the very top in terms of average performance across all metrics.

• It preserved the sharp thermal outlines of people and cars in the dark.
• It kept the fine textures of trees and road markings from the visible images.
• It didn't "wash out" the details or make the images look blurry, a common problem with other methods.

Downstream Tasks:
The paper didn't just stop at making pretty pictures. They tested if these fused images helped robots actually see better.

• Object Detection: They used the fused images to train a YOLOv8s detector (a system that finds objects). CIS-Fuse helped the detector achieve the highest accuracy (mAP of 59.8) compared to all other methods. It was particularly good at spotting people and cars in difficult conditions.
• Semantic Segmentation: They also tested a system that labels every pixel (like coloring in a coloring book). CIS-Fuse achieved the best overall score (mIoU of 74.3), meaning it could accurately outline objects like cars, guardrails, and pedestrians better than the competition.

Energy Efficiency:
This is the "killer feature" of the paper. Because CIS-Fuse uses spiking neurons, it only does work when a spike occurs.

• The researchers measured the "firing rate" of the neurons and found that, on average, only about 14.48% of the neurons fired at any given time.
• Using a standard energy model, they estimated that CIS-Fuse uses roughly one order of magnitude (10 times) less energy than a similarly sized ANN-based method (specifically DCEvo).
• The energy savings get even bigger as the image resolution increases.

What the Paper Rules Out

The paper is very clear about what doesn't work as well as their approach:

• Simple Addition: Just adding the two images together (element-wise addition) or stacking them and running them through a simple filter (concatenation) performed significantly worse. It missed the nuance of how the signals should interact.
• One-Way Fusion: Mixing the signals in only one direction (e.g., Infrared to Visible, but not back) was worse than mixing them both ways. The paper argues that both modalities need to refine each other.
• Symmetric Branches: Using two branches of the exact same length (symmetric) didn't work as well as their asymmetric (shallow vs. deep) design. The paper suggests that the asymmetry is key to getting the network to specialize and learn different roles.
• Simple Neurons: Using older, simpler neuron models (like the standard Integrate-and-Fire or Leaky Integrate-and-Fire without learnable parameters) resulted in worse performance. The paper emphasizes that the ability to learn the "time constant" (how long the neuron remembers the signal) is crucial.

The Bottom Line

The paper CIS-Fuse suggests that we don't have to choose between energy efficiency and high-quality image fusion. By moving the mixing process to the "membrane potential" stage—mixing the signals before the neuron decides to shout—the researchers created a system that is both incredibly efficient and highly effective.

They proved this by showing that their method matches the visual quality of the most advanced, energy-hungry AI models while using a fraction of the power. This is a significant step forward for deploying smart vision systems on battery-powered devices like drones, robots, and autonomous vehicles, where every joule of energy counts. The authors are confident in their results, having tested them across multiple datasets and tasks, and they have made their code available for others to verify and build upon.