💻 computer science

ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval

This paper introduces ChatSearch, a new dataset featuring multi-round multimodal conversational queries for open-domain image retrieval, alongside ChatSearcher, a generative retrieval model trained end-to-end to effectively reason with multimodal context and leverage world knowledge to achieve superior performance in interactive image search.

Original authors: Zijia Zhao, Longteng Guo, Tongtian Yue, Erdong Hu, Shuai Shao, Zehuan Yuan, Hua Huang, Jing Liu

Published 2026-03-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zijia Zhao, Longteng Guo, Tongtian Yue, Erdong Hu, Shuai Shao, Zehuan Yuan, Hua Huang, Jing Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking for a specific photo in a massive, endless library of images. But instead of typing a keyword like "sunset" or "dog," you want to have a conversation with the librarian to find it.

Maybe you start by showing a picture of a beach and saying, "I want something like this, but with a cat instead of a dog." Then, the librarian might ask, "Do you want the cat to be sleeping or playing?" You reply, "Playing, and make it a beach in winter, not summer."

This is the core idea of the paper "ChatSearch." The researchers are building a system that doesn't just search for images based on a single command, but understands a whole back-and-forth chat where the clues are mixed between text and pictures.

Here is a breakdown of their work using simple analogies:

1. The Problem: The "Blind" Librarian

Traditional image search engines are like librarians who only understand one language: either you show them a picture, or you give them a single sentence. They struggle if you try to have a complex conversation.

Example: If you say, "Show me a picture of a car, but make it red," a normal search engine might just show you red cars. It doesn't understand the change you are asking for in the context of a conversation.

2. The Solution: The "ChatSearch" Dataset

To teach computers how to have these conversations, the researchers created a new training dataset called ChatSearch.

The Analogy: Imagine they built a massive "training school" for AI. They didn't just write down questions and answers; they created scenarios.
They used smart AI tools to generate thousands of fake conversations where a user and a computer discuss an image. Sometimes the user shows a picture; sometimes they just talk. The goal of the conversation is always to find a specific "target" image hidden in the database.
They then had human experts check these conversations to make sure they made sense, ensuring the AI learns from high-quality examples.

3. The Star Player: "ChatSearcher"

The researchers also built a new AI model named ChatSearcher. Think of this model as a super-intelligent detective who can read clues from both words and pictures simultaneously.

How it thinks: Unlike old models that treat text and images as separate things, ChatSearcher treats them like a single, flowing story. It can "read" a picture and "listen" to a sentence at the same time.
The Magic Trick (Generative Retrieval): Usually, search engines scan a list and pick the best match. ChatSearcher is different; it generates the answer. It's like a chef who doesn't just pick a dish from a menu but cooks the exact dish you describe.
- When you ask a question, the model predicts the next "token" (a piece of data). Sometimes that token is a word (like "cat"), and sometimes it is a pointer to an image. It literally "writes" the image into the conversation.

4. How It Was Trained (The Two-Stage Workout)

To make ChatSearcher smart, they trained it in two phases:

The "Alignment" Phase: They taught the model to understand that a picture of a dog and the word "dog" are the same thing. They mixed thousands of image-text pairs so the model learned to speak "Image" and "Text" fluently.
The "Conversation" Phase: They gave the model a diet of complex instructions. They showed it how to handle:
- Visual conversations: "What's in this picture?"
- Image editing: "Take this photo and make the sky blue."
- Retrieval: "Find me a photo similar to this one, but with a different background."

5. Why This Matters

The results show that ChatSearcher is much better at finding images through conversation than previous methods.

The "Zero-Shot" Superpower: Even when tested on tasks it wasn't explicitly trained for (like finding images based on a sketch or a complex description), it still performed very well. It's like a student who learns how to study and can ace a test on a subject they've never seen before.
Interactive Refinement: The best part is the interaction. If the AI shows you a picture and you say, "No, I meant the other kind of dog," the AI remembers that context and refines the search immediately.

In a Nutshell

The paper introduces a new way to find images: Stop searching, start chatting.
They created a new dataset (ChatSearch) to teach AI how to understand mixed text-and-image conversations, and they built a new model (ChatSearcher) that acts like a conversational partner who can not only talk about images but also pull the exact right picture out of thin air based on your evolving needs. It turns the cold, mechanical act of "searching" into a warm, natural "conversation."

1. Problem Statement

The paper addresses the challenge of General Conversational Image Retrieval (GCIR) in open-domain settings. Unlike traditional image retrieval tasks that rely on single-round queries (e.g., a single text prompt or a reference image), GCIR requires a system to retrieve images based on multi-round, multimodal conversations between a user and a computer.

Key challenges identified include:

Implicit Intent: Retrieval information is often not explicitly stated but implied within the dialogue context.
Multimodal Reasoning: The system must comprehend and reason across interleaved text and image inputs.
World Knowledge: Successful retrieval often requires leveraging external knowledge to infer user needs that are not directly visible in the provided context.
Lack of Data: There was no existing dataset supporting complex, multi-turn multimodal dialogue for image retrieval.

2. Methodology

The authors propose a two-pronged approach: the creation of a new dataset (ChatSearch) and a novel generative retrieval model (ChatSearcher).

A. The ChatSearch Dataset

The dataset is constructed to simulate realistic, complex retrieval scenarios.

Construction Pipeline:
1. Foundation Models: Utilizes GPT-4 (text generation), CLIP-H (image retrieval), and BLIP-2 (image captioning) to automate data generation.
2. Dialogue Types:
  - MDC-I (Reference Image): Generates single-round dialogues where a user provides a reference image and text instructions to find a target image with specific differences (e.g., background, orientation).
  - MDC-T (Reference Text): Generates two-round dialogues where the first round establishes a visual context via text, and the second round refines the search.
  - Context Merging: Combines plain text and image-text dialogues to create complex, multi-turn interactions.
3. Quality Control: A manual review by domain experts ensures image quality and logical relevance.
Sub-tasks: The dataset is divided into three evaluation splits:
- tChatSearch: Multi-round text-only dialogue.
- iChatSearch: Single-round image-text instruction.
- mChatSearch: Multi-round multimodal dialogue (the most complex).

B. The ChatSearcher Model

ChatSearcher is an end-to-end generative retrieval model based on a causal decoder-only Large Language Model (LLM), initialized with Vicuna-7B.

Architecture:
- Vision Backbone: Uses CLIP ViT-L to extract visual features.
- Perceiver: A Q-former compresses visual embeddings into a fixed number of dense tokens.
- Interleaved Input: Visual tokens are concatenated with text tokens using special [IMG] and [/IMG] markers.
- Dual Heads:
  - Text Head: Predicts the next word in the vocabulary.
  - Visual Head: When the model generates a special [IMG] token, it triggers image retrieval.
Generative Training Objective:
- Treats both word prediction and image retrieval as generative tasks.
- Image Retrieval Mechanism: Uses a dynamically updated feature queue ( $Q$ ) as a "visual vocabulary." The model maximizes the probability of matching the generated query embedding against positive samples in the queue while contrasting against negative samples.
Two-Stage Training Strategy:
1. Stage 1 (Bidirectional Alignment): Pre-trained on interleaved image-text data (CC3M, mmc4-core) to learn image-to-text generation and text-to-image retrieval alignment.
2. Stage 2 (Instruction Tuning): Fine-tuned on a mix of datasets including LLaVA-150k (visual conversation), InstructPix2Pix (image editing), and the ChatSearch training set. This teaches the model to follow complex multimodal instructions and perform retrieval.

3. Key Contributions

ChatSearch Dataset: The first dataset dedicated to general conversational image retrieval, featuring multi-round, multimodal contexts that require reasoning and world knowledge.
ChatSearcher Model: A unified generative model capable of accepting interleaved image-text inputs and producing interleaved image-text outputs (including retrieved images).
Generative Retrieval Paradigm: Demonstrates that treating image retrieval as a generative task (predicting image features from a visual vocabulary) allows for superior reasoning over complex dialogue contexts compared to traditional discriminative retrieval methods.

4. Experimental Results

The model was evaluated on ChatSearch and several zero-shot benchmarks.

General Conversational Image Retrieval (ChatSearch):
- ChatSearcher significantly outperforms baselines (CLIP-t, CLIP-i, CLIP-ti, and FROMAGe) across all sub-tasks (tChatSearch, iChatSearch, mChatSearch).
- Performance: Achieved an average Recall@1 of 27.38% (text), 35.54% (image-text), and 37.90% (multimodal), compared to ~15-16% for the next best baseline.
- Insight: Performance on multi-round multimodal tasks (mChatSearch) was higher than single-round tasks, proving the model's ability to leverage historical context.
Zero-Shot Composed Image Retrieval (CIRR):
- Achieved State-of-the-Art (SOTA) zero-shot performance on the CIRR benchmark, outperforming both zero-shot and fine-tuned specialized models (e.g., CLIP4Cir, CompoDiff).
Zero-Shot Text-to-Image Retrieval:
- On MSCOCO and Flickr30K, ChatSearcher achieved performance comparable to the strong CLIP baseline, validating the effectiveness of the bidirectional alignment training.
Visual Conversation:
- On benchmarks like VQAv2, GQA, MMBench, and SEED-Bench, ChatSearcher achieved performance comparable to SOTA visual dialogue models (e.g., LLaVA-1.5, Qwen-VL), demonstrating it does not sacrifice conversational ability for retrieval capabilities.
Ablation Studies:
- Feature Queue: Increasing the queue size improved performance up to 1000 samples, after which it plateaued.
- Instruction Data: Incorporating visual conversation data (LLaVA) and AIGC editing data (InstructPix2Pix) significantly boosted reasoning and retrieval diversity.

5. Significance and Future Impact

Paradigm Shift: Moves image retrieval from a static "query-response" model to a dynamic, interactive, conversational process.
Reasoning Capability: Demonstrates that LLMs can effectively reason over multimodal contexts to infer implicit user intents, a capability lacking in traditional retrieval systems.
Credible Output: The ability to generate factual, retrieved images alongside text enhances the credibility of AI responses in human-computer interactions.
Future Directions: The authors suggest extending this "fact-based credible output" approach to other modalities like video and audio, paving the way for more sophisticated multimodal agents.

In summary, this work establishes a new benchmark for interactive image retrieval and provides a robust generative framework (ChatSearcher) that successfully bridges the gap between multimodal dialogue understanding and precise visual search.