Beyond Vision: Contextually Enriched Image Captioning with Multi-Modal Retrieval

Problem Statement
Current image captioning systems, despite advancements in multimodal understanding, remain constrained by their reliance on purely visual information. While state-of-the-art models can accurately describe visible objects, they fail to capture crucial contextual details such as event backgrounds, temporal dynamics, named entities, and real-world significance that are not visually discernible. This limitation is particularly detrimental in domains like journalism and digital archives, where images serve as visual documentation requiring rich, paragraph-length descriptions rather than brief, sentence-level summaries. Existing knowledge-enhanced approaches often produce short descriptions focused on immediate visual-textual alignment, failing to provide the comprehensive narratives needed for meaningful event documentation.

Methodology
The authors propose a multimodal pipeline designed to augment visual input with external textual knowledge to generate event-enriched, context-aware captions. The system operates through four primary stages:

1. Multi-Modal Retrieval and Reranking:

   • Semantic Search: The system retrieves semantically similar images and articles using embeddings from BEIT-3 (utilizing Flickr30k-384 and COCO-384) and SigLIP So-384.
   • Ensemble Scoring: To improve robustness, the system combines similarity scores from these models. A unified candidate pool is created from the top results of each model, weighted by position (top-1 vs. top-2) and model contribution. A consensus bonus is applied to candidates appearing in multiple models' top results.
   • Geometric Verification: Retrieved candidates undergo feature matching using ORB and SIFT detectors. The system applies Lowe's Ratio Test and RANSAC to filter matches, computing an inlier ratio, spatial consistency (via a 4x4 grid), and scale consistency. A final confidence score, combining these factors, triggers a rerank if the confidence exceeds 0.4 and minimum inliers are met.

2. Base Caption Generation:

   • The pipeline utilizes InstructBLIP (Vicuna-7B) to generate initial base captions.
   • Instead of generic prompts, a task-specific instruction is employed to explicitly elicit details regarding people, objects, settings, activities, lighting, and atmosphere. This aims to create a structured, factual visual narrative to serve as a bridge for downstream language models, mitigating the "visual bottleneck."

3. Context Extraction and Chunking:

   • Associated news articles are segmented into overlapping text chunks (three sentences with a one-sentence stride).
   • The all-MiniLM-l12-v2 model encodes these chunks and the base caption to perform semantic similarity search. The top five relevant chunks are selected, along with the article's first three and last two sentences to preserve discourse structure.
   • Named entities are extracted via regular expressions to enhance factual grounding.

4. Context-Enriched Caption Generation:

   • A DeepSeek-R1-0528-Qwen3-8B model is fine-tuned using QLoRA (Quantized Low-Rank Adaptation) with a rank of 256 and 8-bit precision.
   • The model is trained using a Causal Language Modeling (CLM) objective. The input consists of the concatenated selected text chunks and a specific prompt instructing the model to prioritize news context (70%) over visual details (30%), emphasize the "5 Ws" (Who, What, Why, When, Where), and produce a 300–350 word caption.
   • The authors note that this semantic filtering approach (selecting relevant chunks) leads to faster convergence and better performance compared to feeding entire articles as context.

Key Contributions and Results
The system was evaluated on the OpenEvents v1 dataset (derived from CNN and The Guardian) and the EVENTA Grand Challenge CondaBench.

• Retrieval Performance: On the private test set, the system achieved an Average Precision (AP) of 0.955, with Recall@1 at 0.945 and Recall@10 at 0.973.
• Captioning Performance:
  • On the public test set, the system achieved a CLIPScore of 0.748, surpassing the best prior baseline (Gemma + Article) by 12.8%.
  • The CIDEr score reached 0.195, which is more than 10 times higher than most prior methods (e.g., Qwen at 0.0282).
  • The system outperformed all evaluated vision-language baselines (including SmolVLM, Qwen, and Gemma) across all key metrics (CLIPScore, CIDEr, BLEU-4, METEOR).

Significance and Claims
The paper claims that this approach successfully transforms image captioning from simple object enumeration into contextually rich, detailed storytelling that bridges visual perception with comprehensive event understanding. By integrating external knowledge via a retrieval-augmented pipeline, the system generates captions that capture named entities, temporal context, and coherent narratives connecting visual content to real-world significance.

The authors acknowledge limitations, noting a reliance on the DeepSeek-Qwen3 model which carries a risk of hallucinations, and a CIDEr score that, while improved, still indicates room for alignment with human phrasing. Future work includes experimenting with alternative base caption generators (such as Qwen-VL 2.5 for better OCR and entity recognition) and exploring post-processing techniques to refine fluency and relevance. The system is positioned as having significant potential for applications in news media, digital archives, and educational platforms where interpretability and semantic depth are essential.