Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models
This paper introduces PeFuse, a training-free framework for Composed Image Retrieval that leverages pretrained Diffusion Models and Multimodal Large Language Models to convert compositional queries into single-modality retrieval tasks via generative conversion, achieving competitive performance without dedicated task-specific training.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine walking through a vast digital library where every book is an image. Traditional search engines let you find a picture by showing them another picture, but they struggle when you want to say, "Show me that same dress, but in red and with long sleeves." This specific challenge, known as composed image retrieval, asks computers to understand a request that mixes a visual reference with a textual instruction. For years, solving this required building custom, complex software that had to be painstakingly trained on massive amounts of labeled data. If the data changed or the request became too specific, these systems often failed. Researchers have long sought a way to make these searches work without needing that heavy, custom training, hoping to create a system that can understand any new request immediately, just by using the powerful tools it already possesses.
A team of researchers at the University of Luxembourg has now demonstrated a way to achieve this without any new training. They developed a method called PeFuse, which acts as a clever translator between the visual and textual worlds. Instead of trying to force a computer to learn how to blend an image and a sentence together from scratch, their system uses two existing, pre-trained artificial intelligence tools to rewrite the request entirely. One tool, a multimodal large language model, is exceptionally good at reading images and writing descriptions. The other, a diffusion model, is a powerful image generator that can create new pictures based on text. The researchers found that by using these tools to convert a mixed request into a single, pure form—either turning the whole request into a new image or turning it into a detailed text description—they could plug the result into standard search engines that were never designed for this task.
The researchers tested this approach on several standard datasets containing fashion items, open-domain photos, and complex scenes. They discovered that the most effective strategy was to convert the mixed request into a text description and then search for images that matched that text. In this scenario, the language model reads the reference photo and the user's instruction, then writes a new, precise sentence that captures exactly what the user is looking for. This new sentence is then fed into a standard image search engine. Surprisingly, this simple translation step worked better than trying to generate a new image from the request and searching for similar pictures. While the image generation tool could create new pictures, those images often contained subtle, unnatural flaws that confused the search engine. The text descriptions, however, remained clear and semantically accurate, allowing the search engine to find the correct matches with high precision.
The study also explored what happens when the system tries to do the reverse: converting the target images in the database into text to match against the user's request. While this worked better than trying to match images against images directly, it still fell short of the text-to-image approach. The researchers found that the quality of the final search result depended heavily on the specific tools chosen. For instance, using a larger, more powerful language model generally improved the results, though the gains were sometimes small. They also examined how different settings for the image generator affected the outcome, finding that using too many steps to generate an image or trying to force the image to look too much like the original reference actually hurt the search performance. The sweet spot was a moderate number of steps and a low setting for how strictly the generator adhered to the original image, allowing it enough freedom to incorporate the new textual instructions.
What makes this work particularly significant is that it requires no new data and no retraining of the underlying models. The system is entirely "training-free," meaning it can be deployed immediately on new datasets or in new domains without the months of preparation usually required. The researchers showed that by simply chaining these pre-existing tools together, they could match or even exceed the performance of complex systems that had been specifically trained for the task. This suggests that the future of this technology may not lie in building bigger, more specialized models, but in finding smarter ways to use the powerful, general-purpose tools we already have. By treating the problem as a translation task rather than a fusion task, the researchers have opened a path for more flexible and adaptable image search systems that can understand human intent without needing to be taught from the ground up.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.