🤖 AI

Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

This paper introduces Re $^3$ Cap, a retrieval-guided reinforcement learning framework that leverages multi-modal retrieval to refine image captions by identifying hallucinations and omissions, thereby outperforming both Supervised Fine-Tuning and existing RL methods like GRPO without requiring additional annotations.

Original authors: Haonan Jia, Shichao Dong, Zenghui Sun, Jiawen Zheng, Ziqi Miao, Gege Shi, Qiuyu Zhao, Jinsong Lan, Xiaoyong Zhu, Bo Zheng

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haonan Jia, Shichao Dong, Zenghui Sun, Jiawen Zheng, Ziqi Miao, Gege Shi, Qiuyu Zhao, Jinsong Lan, Xiaoyong Zhu, Bo Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, computers are learning to see and speak at the same time. These systems, known as large vision-language models, can look at a photograph and describe what is happening in it. This ability is crucial for helping people with visual impairments navigate their surroundings, for organizing vast libraries of images, and for teaching machines to understand the visual world. However, these digital observers are not yet perfect. While they can often identify the main subject of a photo, they frequently invent details that are not there or miss important nuances. They might claim a person is holding a red ball when the ball is actually blue, or they might fail to notice that the sun is casting a long shadow across the ground. This tendency to "hallucinate" or overlook facts makes their descriptions unreliable for serious tasks.

For years, researchers have tried to fix this by showing the computer thousands of examples of correct descriptions, a process similar to how a student learns from a textbook. More recently, another approach has emerged where the computer is allowed to practice on its own, receiving a digital reward when it gets a description right and a penalty when it gets it wrong. This method, called reinforcement learning, has shown promise, but it often struggles to push the computer beyond its initial limits. The computer tends to repeat the same safe, familiar phrases rather than exploring new ways to describe a scene, leaving its potential for deep, accurate observation untapped.

A new study introduces a different way to guide these learning computers, one that relies on the power of comparison rather than just repetition. The researchers, working with a team from Alibaba and other institutions, developed a system called Re3Cap. Instead of asking the computer to guess the right answer in a vacuum, this system gives it a set of similar images to look at first. Imagine the computer is looking at a photo of a tennis player. Before it writes a description, the system finds other photos in its database that look very much like that one. It then reads the descriptions that humans wrote for those similar photos. By comparing its own first draft against this collection of real, human-written descriptions of similar scenes, the computer can spot its own mistakes. If the computer says the player is wearing a hat, but none of the similar photos show hats, the system flags that as a likely error. If the computer misses the fact that the court is green, but every similar photo description mentions the green color, the system knows to add that detail.

The researchers built two specific tools to manage this process. The first tool acts as a careful editor, scanning the descriptions of the similar images to find the facts that appear most often. It tells the computer, "Keep these details because they are consistently true for this kind of scene." The second tool acts as a quality checker. It looks at the difference between what the computer saw in the original photo and what it wrote down. If the computer wrote about a tennis ball that isn't in the picture, or missed a shadow that is clearly visible, this tool identifies those specific gaps. It then feeds this feedback back to the computer, not as a simple grade, but as a set of instructions: remove the invented ball, add the missing shadow, and keep the accurate details about the player's stance.

This process allows the computer to refine its own work before it is finalized. The researchers tested this method on several different types of vision-language models using a standard set of 500 images. They found that by using this retrieval-guided approach, the computers generated descriptions that were significantly more accurate than those produced by previous methods. In tests measuring how well the models understood relationships between objects, the new method improved performance by an average of 8.64 percent compared to the standard reinforcement learning approach. Perhaps most surprisingly, this method worked so well that it outperformed models that had been trained with thousands of human-labeled examples, a much more expensive and time-consuming process.

The study suggests that the key to better image description is not just more data, but a smarter way of using the data that is already available. By letting the computer cross-reference its own observations with a crowd of similar examples, it learns to distinguish between what is actually there and what it merely imagines. The researchers noted that the system is robust, meaning it works well even if the number of similar images it checks changes slightly. However, they also acknowledged that the method depends on having a large and diverse collection of images to draw from; if the library of similar photos is too small, the system cannot find the patterns it needs to correct itself.

Ultimately, this work offers a new path forward for teaching machines to see the world with greater clarity. It shows that by guiding artificial intelligence to look at the world through the lens of similar experiences, we can help it overcome its tendency to make things up. The result is a system that does not just generate words, but constructs descriptions that are grounded in the reality of the image, bringing us closer to a future where computers can truly understand and describe the visual world around us.

Technical Summary: Re3Cap

Problem Statement

While Reinforcement Learning (RL) has demonstrated significant gains in image captioning, it currently faces a critical limitation: it struggles to encourage Large Vision-Language Models (LVLMs) to explore novel reasoning strategies beyond the intrinsic capabilities of the base model. Existing RL approaches often merely select the highest-reward caption from a set of pre-generated candidates, leading to a performance ceiling where RL-optimized models fail to surpass Supervised Fine-Tuning (SFT) methods. Furthermore, current RL-based captioning methods are prone to hallucinations and often fail to capture fine-grained visual details, as they rely on reward signals (such as CLIP scores) that are susceptible to reward hacking due to the constrained compositional reasoning of Vision-Language Models (VLMs).

Methodology

The paper proposes Re3Cap (Retrieval-Guided Refinement for Image Captioning), a retrieval-guided reasoning strategy designed to enhance image captioning without requiring additional annotations. The core insight is that multi-modal retrieval can serve as an effective reasoning signal. Visually similar images share overlapping semantic content; therefore, a detailed and accurate caption should induce retrieval results similar to those obtained from the source image. Discrepancies between image-based and caption-based retrieval results signal misalignments, indicating hallucinations or omissions.

Re3Cap operates through two primary components within an RL framework:

Caption Refinement Suggester (CRS):
- CRS performs image retrieval using the source image as a query against a dataset of image-text pairs.
- It constructs a graph where nodes represent sentences from the retrieved captions. Edges are established based on semantic similarity (using SBERT).
- By computing the k-core subgraph of this graph, CRS filters out long-tail descriptions and identifies semantically consistent elements ( $S^k_v$ ) that frequently appear across similar images.
- These elements serve as a signal to guide the LVLM to incorporate critical, consistent visual details into its generated descriptions.
Caption Quality Assessor (CQA):
- CQA evaluates the quality of a generated caption ( $c$ ) by performing text retrieval using the caption as a query.
- It constructs a graph for the caption-retrieved results and computes its k-core ( $S^k_c$ ).
- A bipartite graph is built between the image-retrieved results ( $S^k_v$ ) and caption-retrieved results ( $S^k_c$ ) to find their intersection ( $S^k_{vc}$ ), representing semantically consistent content shared by both.
- Hallucinations are identified as content present in the caption's retrieval core but absent in the intersection ( $S^k_c - S^k_{vc}$ ).
- Omissions are identified as content present in the image's retrieval core but absent in the intersection ( $S^k_v - S^k_{vc}$ ).

Reinforcement Learning Framework:

Process: The model samples initial captions, generates guidance (correctness, omissions, hallucinations) via CRS and CQA, and then produces refined captions.
Optimization: The framework uses a decoupled PPO formulation. During training, the behavior policy conditions on the image, initial caption, and guidance. However, during policy optimization, the target policy is conditioned solely on the image. This off-policy setting allows the model to learn to generate high-quality captions directly from images at inference time, eliminating the need for retrieval or k-core analysis during deployment.
Objective: The model is optimized to maximize rewards based on the refined captions while minimizing the KL divergence from a reference model.

Key Contributions

Novel Reasoning Strategy: The authors introduce a retrieval-based reasoning strategy that identifies hallucinations and omissions in captions without requiring additional human annotations.
Re3Cap Framework: A reinforcement learning framework that guides LVLMs to generate previously unexplored caption candidates by injecting retrieval-guided reasoning signals, thereby expanding the model's capability boundary.
Performance Superiority: The method demonstrates consistent improvements across various LVLM architectures and reward functions, outperforming state-of-the-art methods, including those relying on Supervised Fine-Tuning.

Experimental Results

Extensive experiments were conducted on COCO-LN500 and DOCCI500 benchmarks using multiple base LVLMs (LLaVA-1.5-7B, Qwen2-VL-7B, Qwen2.5-VL-7B, InternVL3-8B, Qwen3-VL-8B).

Comparison with GRPO: Re3Cap significantly outperforms Group Relative Policy Optimization (GRPO) across multiple reward functions (CLIP, SC, CIM). Notably, on the Relations QA task (a measure of relational reasoning), Re3Cap achieved an average improvement of 8.64% over GRPO on COCO-LN500, and 7.57% on DOCCI500, when averaged across multiple base LVLMs and reward functions.
Comparison with SFT: While standard RL methods often lag behind SFT, Re3Cap outperforms SFT baselines. For instance, using LLaVA-1.5-7B as the base model, Re3Cap surpassed SFT performance across multiple reward functions and benchmarks, whereas GRPO failed to do so.
Reasoning Capability: Analysis using the max@k metric shows that unlike conventional RL which narrows the output distribution, Re3Cap encourages the model to generate diverse, previously unexplored caption candidates, effectively expanding the model's capability boundary.
Ablation Studies: Removing either CRS or CQA resulted in performance drops, confirming that both the identification of critical elements (CRS) and the detection of hallucinations/omissions (CQA) are essential. The combination of both components yielded the best results.
Robustness: The method shows robustness to variations in hyperparameters (retrieval number $K$ and similarity threshold $\tau$ ) and different encoder choices (image and text encoders).
Efficiency: The computational overhead is minimal, requiring only approximately 12.5% more training time compared to GRPO.

Significance

The paper claims that Re3Cap offers a new perspective on improving model performance in image captioning by enhancing the reasoning capabilities of LVLMs within a reinforcement learning setting. By leveraging multi-modal retrieval as a reasoning signal, the method effectively bridges the performance gap between RL and SFT, enabling models to generate more accurate, detailed, and hallucination-free captions without the need for expensive additional annotations. The authors posit that this approach provides valuable insights for future research in both multimodal reasoning and caption generation.