🤖 AI

Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

This paper proposes Cross-modal Identity Mapping (CIM), a reinforcement learning framework that minimizes information loss in image captioning by optimizing for the consistency between an image and its text-retrieved visual counterparts, thereby achieving superior performance without requiring additional annotations.

Original authors: Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Blurry Description"

Imagine you have a very smart robot (a Large Vision-Language Model, or LVLM) that looks at a photo and tries to describe it to you.

Sometimes, this robot is lazy or gets confused.

The Photo: A baseball player in a red helmet swinging a bat at night.
The Robot's Bad Description: "A person is playing a sport with a bat." (Too vague, misses the color, the time of day, and the specific sport).
The Robot's Hallucination: "A cricket player in a white uniform is playing in the sun." (It got the sport wrong and the details wrong).

The paper argues that these mistakes mean the robot is losing information when it converts the picture (visual) into words (text). The goal is to stop the robot from being lazy or making things up.

The Big Idea: The "Reverse Search" Test

The authors came up with a clever way to check if the robot's description is good without needing a human to grade it.

Think of it like a game of "Guess the Photo."

You give the robot a photo and ask it to write a caption.
You take that caption and type it into a search engine (like Google Images).
The Test:
- If the caption is vague ("A person playing sports"), the search engine will show you a messy pile of photos: a tennis player, a soccer goalie, a kid throwing a ball. They don't look alike. The robot failed to be specific.
- If the caption is wrong ("Cricket player in white"), the search engine will show you cricket players. But none of them look like your original photo. The robot made a mistake.
- If the caption is perfect ("Baseball player, red helmet, night time"), the search engine will show you a gallery of photos that all look very similar to each other and very similar to your original photo.

The Insight: If the photos the search engine finds are all consistent with each other and match the original, the description must be detailed and accurate. If the search results are a mess, the description lost information.

The Solution: "Cross-modal Identity Mapping" (CIM)

The authors built a training system called CIM (Cross-modal Identity Mapping) to teach the robot to write better descriptions. They didn't use human teachers; they used the "Reverse Search" test as a teacher.

Here is how the training works, step-by-step:

The Robot Writes: The robot looks at a photo and writes a caption.
The Search: The system takes that caption and searches a huge library of images.
The Score (The Reward): The system gives the robot a score based on two things:
- Consistency (The "Group Hug" Score): Do all the photos found by the search look like they belong to the same family? (e.g., Do they all show baseball players at night?) If yes, the robot gets points for being detailed.
- Relevance (The "Look-Alike" Score): Do the photos found by the search look like the original photo the robot started with? If yes, the robot gets points for being accurate.
The Lesson: If the robot gets a low score, it knows it was too vague or made a mistake. It tries again, adjusting its words to get a higher score next time.

Why This is Special

Usually, to teach a robot to be precise, you need thousands of humans to write perfect descriptions and tell the robot, "Good job" or "Try again." This is slow and expensive.

This paper says: "We don't need humans."
By using the search engine results as a mirror, the robot can teach itself. It learns that to get a high score, it must describe the image so precisely that if you search for it, you can't find anything else that looks different.

The Results

The authors tested this on several different smart robots (like Qwen, LLaVA, and InternVL).

Before: The robots were okay at naming big things (like "a dog") but bad at details (like "a golden retriever with a red collar").
After: Using this "Reverse Search" training, the robots got much better at spotting details and getting the facts right.
The Win: On one specific test, the robot's ability to understand relationships between objects improved by 20%.

Summary

The paper introduces a way to fix "lazy" image descriptions. Instead of hiring humans to check the work, they use a search engine to act as a quality control inspector. If the description is good, the search results will be perfect. If the description is bad, the search results will be messy. The robot learns to write perfect descriptions by trying to make the search results as perfect as possible.

Technical Summary: Cross-modal Identity Mapping (CIM)

Problem Statement

Large Vision-Language Models (LVLMs) frequently fail to generate fine-grained, accurate image captions, often omitting or misrepresenting critical visual details. This "information loss" during modality conversion (image-to-text) hinders downstream tasks such as visual question answering and image-text retrieval.

Existing approaches to mitigate this issue face significant limitations:

Supervised Fine-Tuning (SFT): Requires costly, time-consuming human annotation of detailed image descriptions, which is difficult to scale.
Reinforcement Learning (RL) with Metrics: Previous methods attempt to quantify information loss using metrics like keyword matching or VLM-based analysis. However, these often lack explicit supervision for non-keyword elements or suffer from "reward hacking" due to the limited compositional reasoning abilities of current VLMs.

Consequently, accurately quantifying information loss in image captioning without additional human annotations remains a challenging, underexplored problem.

Methodology

The paper proposes Cross-modal Identity Mapping (CIM), an annotation-free reinforcement learning framework. The core hypothesis is that the quality of an image caption is positively correlated with the semantic consistency and relevance of images retrieved using that caption as a query.

1. Quantitative Evaluation Metrics

CIM introduces two metrics to quantify information loss by analyzing the distribution of retrieved images:

Gallery Representation Consistency (GRC): Measures the internal consistency of images retrieved by a caption. High GRC indicates that the caption captures fine-grained details, resulting in a gallery of visually similar images. It is calculated using the mean resultant length of image embeddings in the support set.
Query-gallery Image Relevance (QIR): Measures the relevance between the source image and the retrieved gallery. High QIR indicates that the caption accurately describes the specific visual content of the source image. It is calculated as a weighted sum of cosine similarities between the source image and retrieved images.

2. Verification of Information Loss

Before applying the method, the authors verified the existence of information loss across various LVLMs (e.g., Qwen3-VL, InternVL) using the Oxford-IIIT Pet dataset. They demonstrated that while models achieve high accuracy in coarse-grained species classification, their performance drops significantly in fine-grained breed classification when relying solely on generated captions. This confirms that LVLMs often lose fine-grained visual details during text generation.

3. Reinforcement Learning Framework

CIM utilizes these metrics as a reward signal to optimize LVLMs via Group Relative Policy Optimization (GRPO):

Reward Function: $\Upsilon(v, c) = GRC(c) + \beta \cdot QIR(v, c)$ , where $v$ is the image, $c$ is the caption, and $\beta$ balances precision and detail richness.
Process:
1. The LVLM samples a group of captions for a given image.
2. Each caption is used to retrieve the top- $K$ relevant image-text pairs from a corpus.
3. GRC and QIR are computed based on the retrieved gallery.
4. The advantage is calculated relative to the group mean, and the model is updated to maximize this reward.
Goal: The framework forces the LVLM to minimize information loss, effectively achieving an "identity mapping" where the text description preserves the visual identity of the source image.

Key Contributions

Annotation-Free Metric: The paper converts the quantification of caption quality into a measure of image-to-image similarity, introducing GRC and QIR to evaluate information loss without human annotations.
Cross-modal Identity Mapping (CIM): A novel RL framework that uses the proposed metrics as rewards to encourage LVLMs to generate fine-grained and precise captions.
Empirical Validation: Extensive experiments demonstrate that CIM improves performance across various LVLM architectures (LLaVA, Qwen-VL, InternVL) and scales, outperforming state-of-the-art methods including Supervised Fine-Tuning and SC-Captioner.

Experimental Results

The method was evaluated on COCO-LN500 and DOCCI500 benchmarks using multiple LVLMs (e.g., LLaVA-1.5, Qwen2/2.5-VL, InternVL2/2.5/3).

Performance Gains: CIM consistently improved caption quality across all metrics (CAPTURE, Objects, Attributes, Relations).
- On Qwen2.5-VL-7B with the COCO-LN500 benchmark, CIM achieved a 20.4% improvement in the QA score for Relations and a 20.2% improvement in the F1 score for Attributes compared to the base model.
- Compared to the base model, the method showed an average improvement of 3.7% on the CAPTURE metric.
Comparison with SFT: Even when applied to models already fine-tuned with SFT (on RefinedCaps), CIM provided further improvements, particularly in challenging sub-tasks like Attributes and Relations, outperforming the SC-Captioner baseline.
Ablation Studies: Both GRC and QIR were shown to be effective individually, but their combination yielded the best performance, confirming their complementary nature.
Scalability & Robustness:
- Performance improved as the retrieval corpus size increased (from 6.5K to 1M images), suggesting that larger corpora force the model to generate more precise captions to retrieve relevant images.
- The method demonstrated strong robustness across different image and text encoders (e.g., OpenCLIP, DINOv3, MPNet, MiniLM), with minimal performance variance.

Significance and Claims

The paper claims that CIM provides a new perspective for improving LVLM image captioning by shifting the focus from direct text supervision to cross-modal consistency. By leveraging the retrieval process as a verifiable reward signal, the method forces models to attend to richer visual details and generate precise descriptions without the burden of expensive human annotation.

The authors assert that this approach successfully minimizes information loss during modality conversion, enabling LVLMs to establish a more faithful identity mapping from visual inputs to textual outputs. The work highlights that stronger base models benefit more from this RL process, suggesting a scalable path toward higher-quality multimodal generation.