Vision-Modulated Soft-Part-Aware Mamba for Video-Based Visible-Infrared Person Re-Identification

In the quiet hum of a city's surveillance network, cameras watch constantly, capturing the flow of people day and night. For decades, security systems have relied on visible-light cameras, the same kind that capture color photographs, to identify individuals by their clothing and features. But when the sun sets and the light fades, these cameras struggle; the colors wash out, and textures blur into indistinguishable shadows. To solve this, security teams increasingly pair visible cameras with infrared sensors, which see the heat radiating from a person's body rather than the light reflecting off their clothes. This creates a new challenge: how does a computer recognize that a person seen in a grainy, heat-based infrared video is the same person seen in a clear, color video from a different camera? This task, known as visible-infrared person re-identification, requires a system to ignore the massive differences in how the two cameras see the world and focus instead on the unique identity of the person walking through the frame.

The difficulty deepens when the subject is moving. A person is rarely a static image; they are a sequence of movements, a walking gait that shifts with every step. Existing computer systems often try to solve this by looking at the whole person at once or by cutting the image into rigid, fixed strips, like slicing a loaf of bread. However, people do not walk in straight lines; they twist, turn, and lean. A rigid slice might capture a person's head in one frame and their shoulder in the next, breaking the continuity of the identity. Furthermore, the way a person looks in the dark is fundamentally different from how they look in the light, making it hard for a computer to match the same body part across these two different views. Researchers at Changzhou University have proposed a new approach to this problem, one that abandons rigid slicing in favor of a more fluid, adaptive method that learns to track specific body regions as they move and change.

The researchers developed a system they call VM-SPMamba, which operates by first understanding the entire scene and then focusing on specific, flexible parts of the person. Instead of forcing the computer to look at fixed horizontal strips of the image, the system learns to identify "soft parts." Imagine a computer that can decide, on its own, where a person's head, torso, and legs are located in each frame, even if the person is leaning or if the camera angle shifts. The system does this by using a global view of the person to guide its attention, allowing it to group pixels into meaningful regions that adapt to the person's pose. This means that if a person raises their arm, the system adjusts the boundaries of the "arm" region to follow the movement, rather than sticking to a pre-set box that might cut the arm in half.

Once these flexible regions are identified, the system treats each one as its own independent story. It tracks the "head" region across all the frames of the video, then tracks the "torso" region across the same frames, and so on. By separating the video into these distinct streams of movement, the system can learn how each specific body part evolves over time. This is crucial because the way a head moves is different from the way a leg moves, and these differences are key to identifying a person. The researchers found that by modeling these parts separately and then combining their stories, the system becomes much better at ignoring the confusing differences between the infrared and visible cameras. It learns that a specific pattern of movement in the infrared view corresponds to the same pattern in the visible view, even if the colors and textures are completely different.

To teach the system this skill, the researchers used a powerful training method that involves a "language" component. They fed the system not just images, but also text descriptions of identities, allowing the computer to link the visual patterns of a person's movement to a semantic concept of who that person is. This acts as a bridge, helping the system understand that the heat signature of a person walking is the same identity as their color image, even though they look nothing alike. The system was tested on two large datasets containing hundreds of thousands of images from real-world surveillance scenarios. In the infrared-to-visible test, where the system tries to find a person in a color video after seeing them in a heat-based video, the new method achieved a success rate of 72.3 percent. In the reverse direction, it reached 72.4 percent. These numbers represent a significant leap forward, surpassing previous best results by more than ten percentage points in some measures.

The researchers also tested their system on a much larger and more complex dataset with over three thousand different people and longer video sequences. While the system did not achieve the absolute highest score on every single metric in this more difficult test, it remained highly competitive, correctly identifying the right person as the top match in nearly 70 percent of the infrared-to-visible cases and 70 percent of the visible-to-infrared cases. This suggests that the method is robust enough to handle the messy, unpredictable nature of real-world surveillance, where people walk in crowds, change direction, and appear in varying lighting conditions. The study explicitly showed that older methods, which rely on fixed, unchanging slices of the image, perform worse when people move or when the camera angles shift. By proving that flexible, adaptive parts are superior to rigid ones, the research offers a clearer path for building surveillance systems that work reliably day and night.

The success of this approach lies in its ability to mimic the way humans might intuitively track a person: by focusing on the flow of their body parts rather than a static snapshot. The system does not need to be told where the head or the legs are; it learns to find them by observing how the pixels move together. This adaptability allows it to handle the distortions caused by walking, turning, and the different ways cameras capture the world. While the current system still relies on labeled data to learn these patterns and uses a fixed number of body parts, the results demonstrate that moving away from rigid structures toward fluid, content-aware representations is a powerful strategy. As surveillance networks continue to expand and rely more on infrared technology for nighttime security, methods like this one provide the necessary tools to ensure that the right person is identified, regardless of the time of day or the camera capturing the scene.