CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification

Imagine you are trying to find a specific friend in a crowded city, but you have two very different maps. One map is a vibrant, high-definition color photo taken during the day, showing your friend's bright red jacket and blue sneakers. The other map is a grainy, black-and-white thermal image taken at night, where your friend looks like a glowing white blob against a dark background. This is the daily challenge for security cameras and computer vision systems: matching a person seen in the daylight (visible light) with the same person seen in the dark (infrared). While humans can easily connect the dots between a colorful face and a heat signature, computers struggle because the "gap" between these two types of images is massive. They look nothing alike, making it hard for standard AI to realize they are looking at the same person. To solve this, researchers often use a clever trick involving a "universal translator" called CLIP. Think of CLIP as a super-smart librarian who has read millions of books and seen millions of photos; it knows that the word "sneakers" connects to the picture of shoes, even if the picture is blurry or taken in the dark. By using words as a bridge, the computer can try to match the day photo to the night photo through their shared description.

However, there is a catch. The "librarian" (CLIP) was trained mostly on colorful, sunny photos. If you ask it to describe a glowing heat blob from a night camera, it often gets confused, producing vague or noisy descriptions like "a vague person" instead of "a person with a backpack." This confusion throws off the matching process. In this paper, the authors propose a new method called CLIP4VI-ReID to fix this problem. Instead of asking the confused librarian to describe the night photo directly, they first ask it to describe the clear day photo. Then, they use that accurate description as a guide to "teach" the computer how to see the night photo correctly. They do this in three steps: first, they generate perfect text descriptions from the day photos; second, they use those words to correct the computer's understanding of the night photos; and finally, they fine-tune the whole system so that the day photos, night photos, and the text descriptions all line up perfectly. The results show that this step-by-step approach is much better at finding the right person than previous methods, proving that using the right kind of "words" as a bridge can help computers see clearly, even in the dark.

The Problem: The Great Modality Gap

Imagine you are trying to match a suspect's ID card (a clear, color photo) with a security camera feed from a dark alley (a thermal, black-and-white image). In the world of computer vision, this is called Visible-Infrared Person Re-identification (VI-ReID). The goal is simple: find the same person across these two very different types of images. But it's incredibly hard because the two images look nothing alike. The color photo is full of details like red shirts and blue jeans, while the thermal image is just a glowing shape based on body heat.

For a long time, computers tried to solve this by either trying to turn the thermal image into a color one (which often looks fake and noisy) or by trying to find common features directly between the two. But the authors of this paper noticed a flaw in how newer, smarter AI models (based on something called CLIP) were handling this. These models tried to generate a text description for both the color photo and the thermal photo. The problem? The AI was trained on millions of sunny, colorful photos. When it looked at a thermal image, it didn't know what to say. It would generate weak, inaccurate descriptions like "a hard-to-see person" instead of useful details like "wearing a hat." This bad description made it harder for the computer to match the person, rather than easier.

The Solution: A Three-Stage Bridge

To fix this, the authors built a new system called CLIP4VI-ReID. Instead of forcing the AI to guess what the thermal image looks like in words, they designed a clever three-stage process that acts like a construction crew building a bridge between the two worlds.

Stage 1: The Text Semantic Generation (TSG)
First, the team decided to stop asking the AI to describe the thermal image. Instead, they only asked it to describe the clear, color photos. They used a technique called "prompt learning," which is like giving the AI a fill-in-the-blank sentence: "A photo of a [_____] person." The AI learns to fill in the blanks with specific details (like "wearing sneakers" or "carrying a backpack") based only on the color photos. Because the color photos are clear, the AI generates perfect, detailed text descriptions. These descriptions become the "gold standard" or the bridge.

Stage 2: Infrared Feature Embedding (IFE)
Now that they have the perfect text descriptions from the color photos, they use them to teach the AI how to look at the thermal images. They don't ask the AI to generate new words for the thermal image. Instead, they tell the AI: "Look at this thermal image, and try to make its features match the text we already wrote for the color photo." This is like showing a student a clear map and then asking them to navigate a foggy path using that same map as a guide. The AI learns to adjust its understanding of the thermal image so that it aligns with the accurate text description. This step "rectifies" or fixes the thermal image features, injecting the correct identity information into them.

Stage 3: High-level Semantic Alignment (HSA)
Finally, the team fine-tunes the whole system. They make sure that the text descriptions don't accidentally include details that are only in the color photo (like specific colors) and aren't shared with the thermal photo. They want the text to describe only the things that are true for both images (like "a person with a backpack"). They also tweak the parts of the AI that look at the color and thermal images separately to make sure they are picking up the best details. This final stage ensures that the color photo, the thermal photo, and the text description all point to the same person with high precision.

What They Found

The authors tested their new system on two famous datasets of people in color and thermal images: SYSU-MM01 and RegDB. They compared their method against the best existing techniques.

• On the SYSU-MM01 dataset: Their method, CLIP4VI-ReID, achieved a Rank-1 accuracy of 75.54% in the "All-Search" setting (where the system has to find the person among many possibilities). This was better than the previous best method, which got 75.2%. In the "Indoor-Search" setting, they achieved an even higher 83.98% Rank-1 accuracy, beating the next best method by a significant margin.
• On the RegDB dataset: The results were even more impressive. In the "Infrared-to-Visible" test (finding a color photo from a thermal query), they reached a Rank-1 accuracy of 92.28%, which was the highest among all tested methods. In the "Visible-to-Infrared" test, they hit 94.51% Rank-1 accuracy.

The paper also included visual proof. They showed that before their method, the computer's "mental map" of color and thermal images were in completely different places. After their three-stage process, the images of the same person (whether color or thermal) clustered tightly together, while images of different people stayed far apart.

Why It Matters

The key takeaway is that trying to force a computer to describe a thermal image directly often leads to confusion and errors. By using the clear color image to generate the description first, and then using that description to guide the thermal image, the system avoids the "noise" and gets much better results. The authors suggest that this "coarse-to-fine" approach—starting with a rough alignment and then refining it—is a powerful way to help computers see across different types of light. While the method requires a bit more training time than simpler models, the improvement in accuracy suggests it's a worthwhile trade-off for security systems that need to work 24/7, day or night.