Robust RGB-IR Object Detection under Missing-Modality Conditions via Complementary Feature Restoration and Cross-Modal Fusion

In the world of machine vision, computers are taught to see the world much like humans do, using cameras that capture visible light. These systems are the eyes behind self-driving cars, security cameras, and automated factories. However, just as human eyes struggle in the pitch black of a moonless night or through thick fog, these digital eyes often fail when the light fades or the weather turns bad. To solve this, engineers have long paired standard color cameras with infrared sensors. While a standard camera sees color and texture, an infrared camera sees heat, allowing it to spot a person or a car even in total darkness. By combining these two views, a system can see more clearly than either could alone. But this solution has a hidden weakness: it assumes both cameras will always work perfectly. In the real world, sensors break, lenses get dirty, or signals get lost. If one camera fails, the entire system often collapses, leaving the machine blind.

A team of researchers from Taiwan has developed a new way to build these vision systems so they can keep working even when one of their eyes goes dark. Their approach, detailed in a recent study, focuses on creating a computer model that doesn't just wait for two perfect images but actively tries to fix the picture when one is missing or damaged. Instead of treating a broken sensor as a total failure, the new system acts like a skilled restorer, using the information from the working camera to fill in the gaps left by the broken one. This allows the machine to maintain its ability to spot objects, whether it is a pedestrian on a dark street or a vehicle on a rainy highway, even when it only has half the data it usually relies on.

The researchers tested their idea using two large collections of image pairs: one containing thousands of scenes with cars, people, and bicycles in various lighting conditions, and another focused specifically on finding people in extremely dark urban environments. They trained their system to handle three different scenarios: when both cameras worked, when one camera was completely missing, and when the system had to guess which camera was working for different parts of the test. In the ideal situation where both cameras provided clear images, the new system correctly identified objects in 80.2% of the cases. This performance was competitive with the best existing methods, proving that the new design did not sacrifice accuracy for the sake of resilience.

The true test, however, came when the researchers simulated failures. They deliberately removed the input from one camera to see how the system coped. When the system was forced to rely on only the infrared camera, it managed to find objects with high accuracy, achieving a success rate of 78.2%. When it had to rely solely on the standard color camera, it still performed well, correctly identifying objects in 63.7% of the cases. Perhaps most impressively, when the researchers mixed the conditions so that half the test images lacked a color camera and the other half lacked an infrared camera, the system maintained a success rate of 73.5%. This stability stood in sharp contrast to older systems, which saw their performance drop dramatically or fail entirely when one of their inputs was removed.

To achieve this, the researchers built a framework with two main innovations. First, they created a module designed to repair the features of the missing or degraded image. Imagine a painter who, upon noticing a hole in a canvas, uses the colors and patterns from the surrounding area to reconstruct the missing piece. Similarly, this computer module analyzes the working camera's data to infer what the missing camera should have seen, effectively restoring the lost information before the two images are combined. Second, they improved the way the system merges these two different types of vision. Instead of simply stacking the images on top of each other, the system uses a sophisticated attention mechanism to focus only on the most relevant parts of the image, ignoring the noise and confusion that often arise when trying to mix heat signatures with color details. This ensures that the final decision is based on the clearest possible signal.

The study suggests that this method offers a significant step forward for practical applications where reliability is more important than peak performance in perfect conditions. In real-world settings, such as a security camera on a street corner or a sensor on a delivery drone, equipment failure is a matter of time, not possibility. A system that can gracefully handle a broken sensor is far more valuable than one that delivers perfect results only when everything is working. By demonstrating that a machine vision system can remain robust even when half its sensory input is gone, the researchers have provided a blueprint for building safer, more dependable technology for the unpredictable world outside the laboratory.