Seeing Through Words: Controlling Visual Retrieval Quality with Language Models

This paper proposes a quality-controllable text-to-image retrieval framework that leverages generative language models to enrich short, ambiguous user queries with contextual details and explicit quality constraints, thereby improving retrieval performance and enabling steerable, transparent results without modifying existing vision-language models.

The Gist

Imagine you are walking into a massive, endless library of photos. You want to find a picture of a "dog."

The Problem:
You shout, "Dog!" to the librarian.
Because the library has millions of dogs, the librarian pulls out a chaotic pile: a sleeping puppy, a fierce guard dog, a dog in a costume, a blurry photo of a dog, and a high-definition, artistic portrait of a dog.
You wanted something specific (maybe a cute, high-quality photo), but because your request was so short ("dog"), the librarian had to guess. You get a mix of everything, and you have to sift through the bad ones to find the good one.

The Paper's Solution:
This paper proposes a new way to talk to the librarian. Instead of just shouting "Dog," you use a smart assistant (an AI language model) to help you finish your sentence before you ask.

Here is how it works, broken down with simple analogies:

1. The "Magic Translator" (Query Completion)

Think of your short query ("dog") as a rough sketch. The system uses a Generative AI (like a very smart writer) to turn that sketch into a detailed painting description.

• You say: "Dog."
• The AI thinks: "Okay, the user wants a dog. But what kind of dog? And how good should the picture look?"
• The AI expands your request to: "A golden retriever puppy playing in a sunny park, with soft lighting and a professional photography style."

2. The "Quality Dial" (Controllability)

This is the paper's secret sauce. Usually, AI just tries to make the description longer. But this system adds a dial that lets you choose the quality of the result.

Imagine the AI has a menu of options for you to pick from:

• Setting A (Low Quality): The AI expands "dog" to: "A blurry, grainy photo of a dog seen through a fence." (Maybe you just want a quick, rough idea).
• Setting B (Medium Quality): The AI expands it to: "A dog running in a park, taken with a standard camera."
• Setting C (High Quality): The AI expands it to: "A stunning, high-definition portrait of a dog with golden hour lighting, sharp focus, and artistic composition."

You can tell the system, "I want Setting C," and it will rewrite your request specifically to find those high-quality images.

3. Why This is Better Than Just Filtering

You might ask, "Why not just search for 'dog' and then throw away the bad pictures?"

• The Old Way (Filtering): Imagine the librarian brings you 1,000 photos of dogs. You have to look at all 1,000 to find the 5 good ones. It's slow and frustrating.
• The New Way (Quality Control): The librarian only brings you the 5 best photos because you told them exactly what you wanted before they started looking.

The Three Superpowers of This System

The authors highlight three main benefits:

1. Flexibility (The Universal Adapter): It works with any existing photo-search engine. You don't need to rebuild the whole library; you just add this smart "translator" in front of it.
2. Transparency (The Honest Assistant): The AI doesn't just magically change the results. It shows you the new sentence it wrote ("A golden retriever..."). You can read it, understand it, and say, "Oh, I didn't want a golden retriever, I wanted a poodle," and change it.
3. Controllability (The Volume Knob): You aren't stuck with whatever the computer thinks is "best." You can turn the knob to "High Aesthetics" for art projects or "High Relevance" if you just need to find a specific object quickly.

The Bottom Line

This paper solves the problem of "short, vague searches" by teaching the computer to ask better questions for you.

Instead of you struggling to describe exactly what you want, the AI acts like a creative writing partner. It takes your one-word idea and expands it into a detailed, high-quality instruction that guarantees the photos you get back are exactly the kind of quality you are looking for. It bridges the gap between your simple thought and the computer's complex database.

Technical Summary

1. Problem Statement

The paper addresses a critical limitation in Text-to-Image Retrieval (T2IR): the degradation of performance when users provide short, underspecified queries (e.g., "a dog").

• Semantic Ambiguity & Collisions: Short queries map to a vast, diffuse region in the embedding space, causing semantic collisions where visually diverse images receive similar similarity scores.
• Lack of Quality Control: Existing systems focus solely on semantic alignment. They cannot explicitly steer retrieval results toward specific user preferences regarding image quality, such as aesthetics (visual appeal) or relevance (semantic consistency).
• The Gap: There is a disconnect between the expressive capacity of modern Vision-Language Models (VLMs) and the limited information provided by short user queries.

2. Methodology: Quality-Conditioned Query Completion (QCQC)

The authors propose a new paradigm called Quality-Controllable Retrieval (QCR). Instead of modifying the frozen VLM backbone, they enrich the input query using a generative Large Language Model (LLM) conditioned on discretized quality levels.

A. Core Framework

The system operates in three stages:

1. Data Annotation & Discretization:
   • For a given image gallery, the system generates textual descriptions (using a captioning model), relevance scores (via VLM cosine similarity), and aesthetic scores (via an aesthetic predictor).
   • Continuous scores are discretized into categorical levels (e.g., Low, Medium, High) based on percentiles.
2. Training the Query Completion Function:
   • A generative LLM is fine-tuned to act as a query completion function $h(Q)$.
   • Input: A short query $Q$ + an instruction specifying the desired quality condition $C$ (e.g., "Relevance: High, Aesthetic: High").
   • Output: An enriched query $Q^+$ that appends descriptive details (pose, scene, lighting, context) tailored to the specified quality level.
   • Loss: Standard autoregressive next-token prediction loss.
3. Inference:
   • Given a short user query, the system generates multiple enriched versions based on different quality constraints.
   • These enriched queries are fed into a frozen, pre-trained VLM (e.g., OpenCLIP) to retrieve images.
   • The retrieval is performed via standard similarity search (e.g., FAISS) on the image embeddings.

B. Theoretical Justification

The paper provides a theoretical analysis modeling query completion as a structured perturbation of the text embedding matrix.

• Using Singular Value Decomposition (SVD), the authors prove that under certain conditions, enriching queries increases the rank of the resulting score matrix ($S_B$) compared to the original matrix ($S_A$).
• A higher rank implies the system can express more independent scoring patterns, allowing for finer-grained distinctions between images and reducing semantic collisions.

3. Key Contributions

1. New Problem Setting: Introduction of Quality-Controllable Retrieval (QCR), a framework where retrieval is explicitly conditioned on user-defined quality dimensions (Relevance and Aesthetics).
2. General Solution (QCQC): A flexible framework that leverages LLMs to enrich short queries. It is VLM-agnostic, meaning it works with any frozen pre-trained VLM without requiring retraining of the heavy vision encoder.
3. Three Key Advantages:
   • Flexibility: Compatible with any existing VLM.
   • Transparency: The enriched queries are human-readable, allowing users to inspect and understand why certain images were retrieved.
   • Controllability: Users can explicitly steer results toward "High Aesthetic" or "High Relevance" without post-hoc filtering.

4. Experimental Results

The method was evaluated on two datasets: MS-COCO (with human annotations) and Flickr2.4M (constructed from Openverse).

• Baselines: Compared against direct prefix usage, pre-trained LLMs (without fine-tuning), and fine-tuned LLMs with random quality labels.
• Performance:
  • Quality Control: QCQC successfully steers retrieval. As the quality condition moves from Low $\to$ High, both average aesthetic and relevance scores of retrieved images increase significantly.
  • Comparison to Baselines:
    • Pre-trained LLMs (without fine-tuning) often degrade performance by adding irrelevant noise.
    • Fine-tuning without quality conditioning improves relevance but fails to control aesthetics.
    • QCQC outperforms all baselines in both metrics while maintaining control.
  • Granularity: The method works effectively even with 5 distinct quality levels (Very Low to Very High).
  • Post-Retrieval Filtering: QCQC outperforms a two-stage baseline (retrieve top-k by relevance, then re-rank by aesthetics). The paper argues that short queries lack the semantic density to support effective re-ranking; QCQC fixes the problem at the query generation stage.

5. Significance and Impact

• Bridging the Gap: The work effectively bridges the gap between the high capacity of modern VLMs and the "underspecified" nature of real-world user queries.
• User-Centric Retrieval: It shifts the retrieval paradigm from purely semantic matching to user-centric quality control, acknowledging that "best" results depend on context (e.g., an architect needs unique/creative images, while a shopper wants popular/aesthetic ones).
• Efficiency: By keeping the VLM frozen and only training a lightweight LLM for query expansion, the approach is computationally efficient and easily deployable.
• Future Directions: The authors suggest extending this to other quality dimensions like "interestingness," "diversity," and "personalization."

In summary, QCQC transforms short, ambiguous queries into rich, quality-aware descriptions, enabling precise control over the visual characteristics of retrieved images without altering the underlying retrieval models.