Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension

Technical Summary: Chain-of-Caption: Training-Free Improvement of Multimodal Large Language Model on Referring Expression Comprehension

Problem Statement
Referring Expression Comprehension (REC) is a longstanding challenge in vision-language learning, requiring a model to localize a specific object within an image based on a textual description. While Multimodal Large Language Models (MLLMs) have achieved high accuracy (>80% at 0.5 Intersection over Union (IoU)) on benchmarks like RefCOCO, they struggle with high IoU thresholds (>0.7). Existing models often fail when the target object is in the background, partially occluded, or when multiple instances of the same class exist. Furthermore, current approaches often mix various context-providing tools without isolating the specific contribution of individual components, leaving the source of performance gains unclear.

Methodology
The authors propose Chain-of-Caption, a training-free framework designed to enhance the REC performance of generalist MLLMs by leveraging their multi-task capabilities (grounded description generation, Visual Question Answering (VQA), and image captioning) to iteratively refine predictions. The methodology operates through the following stages:

1. Context Augmentation: The framework introduces both textual and visual contexts.

   • Textual Context (Grounded Description): The MLLM is prompted to detect a specified number of objects in the image, outputting a numbered list containing object descriptions and their corresponding bounding box coordinates. This "grounded description" serves as a bridge between the textual query and the visual scene.
   • Visual Context: The framework utilizes image cropping around an initial predicted region to focus the model's attention, and visualizes bounding boxes to provide spatial guidance.

2. Iterative Refinement (Chain-of-Caption):

   • Initialization: The model generates an initial grounded description ($g$) and an initial bounding box prediction ($b$) for the target text ($t$).
   • Verification Loop: The model employs its VQA capability to verify if the cropped region defined by $b$ matches the target description, returning a binary "yes" or "no" answer.
   • Correction: If the answer is "no," the model generates a caption ($c$) for the failed cropped region. This caption, along with the previous bounding box, is appended to the grounded description to update the context.
   • Re-prediction: The model uses this enriched context to generate a new bounding box prediction ($b_{CoC}$). This process repeats until the verification returns "yes" or a maximum number of trials is reached.

Key Contributions

• Analysis of Contextual Tools: The paper systematically evaluates the individual effects of textual contexts (object descriptions vs. grounded descriptions) and visual contexts (cropping vs. drawing bounding boxes) on REC performance.
• Training-Free Framework: Chain-of-Caption improves MLLM performance without any fine-tuning, relying solely on prompt engineering and the model's inherent multi-task abilities.
• Iterative Grounding: The framework uniquely combines grounded descriptions with a self-correction loop using VQA and captioning to handle ambiguous or difficult localization tasks.

Experimental Results
Experiments were conducted on RefCOCO, RefCOCO+, RefCOCOg, and the more challenging Ref-L4 datasets using NVILA-8B, NVILA-15B, and Qwen2.5-7B models.

• Impact of Context: The study found that while simple object descriptions had minimal impact, grounded descriptions significantly improved accuracy at high IoU thresholds (0.7 and 0.9), though they slightly reduced performance at 0.5 IoU. Cropping improved high-threshold accuracy but degraded low-threshold performance.
• Performance Gains: Chain-of-Caption demonstrated substantial improvements over baseline models. On the NVILA-8B model, it achieved a 5% to 30% performance gain across various IoU thresholds. Specifically, on Ref-L4, the method improved Acc0.7 from 30.28% to 56.63% (+26.35%) and Acc0.9 from 2.82% to 25.16% (+22.34%).
• Model Scaling: The framework proved effective across different model sizes. Notably, while NVILA-15B underperformed NVILA-8B in its base form, Chain-of-Caption enabled NVILA-15B to surpass the smaller model, suggesting superior in-context learning capabilities in larger models.
• Limitations: The method relies on the model's ability to consistently follow formatting instructions (e.g., JSON output for bounding boxes). The authors noted that Qwen2.5-7B struggled with consistent output formatting in its base form, requiring specific prompting strategies to ensure reliability.

Significance
The paper claims that Chain-of-Caption substantially improves bounding box detection accuracy, particularly at higher IoU thresholds (>0.7), resulting in better-aligned bounding boxes. By isolating the effects of different context types, the authors demonstrate that grounding textual concepts with bounding box coordinates is the most effective mechanism for improving REC. The work highlights that existing MLLMs possess the latent capability to refine their own predictions through iterative context generation and verification, offering a path to high-precision localization without the computational cost of retraining.