💻 computer science

SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrieval

SeCo-SBIR is a semantically consistent prompt learning framework that resolves the tension between domain adaptation and zero-shot generalization in sketch-based image retrieval by injecting text-guided semantic knowledge into the visual encoder and enforcing consistency with a frozen CLIP reference via perturbation-based constraints, achieving state-of-the-art performance across multiple benchmarks.

Original authors: Long Hoang Dang, Tuan Nguyen Huu, Nguyen Minh Hieu, Tu Minh Phuong

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Long Hoang Dang, Tuan Nguyen Huu, Nguyen Minh Hieu, Tu Minh Phuong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to recognize things, but with a twist: you can only show it photos of real animals, yet you want it to understand drawings made by humans. This is the world of Zero-Shot Sketch-Based Image Retrieval. It's a branch of artificial intelligence where a computer has to match a rough, scribbled sketch (like a stick figure dog) to a high-definition photograph of a real dog, even if it has never seen that specific type of dog before.

To understand the challenge, think of the robot's brain as having two main "senses": one for seeing (photos) and one for reading (text). A powerful tool called CLIP has already taught this robot to understand that the word "dog" and a photo of a dog are related. However, when you ask the robot to look at a messy, black-and-white sketch, it gets confused. The sketch is too different from the photo. The usual fix is to "fine-tune" the robot, teaching it new tricks to bridge the gap. But here's the catch: if you teach it too much about the specific dogs you show it during training, it gets stuck in a rut. It becomes so obsessed with the training examples that it forgets how to recognize new types of dogs it hasn't seen yet. It's like a student who memorizes the answers to a practice test so perfectly that they fail the real exam because the questions are slightly different.

This is exactly the problem the researchers at the Posts and Telecommunications Institute of Technology set out to solve. They introduced a new method called SeCo-SBIR, which acts like a clever translator and a strict teacher rolled into one. Instead of just forcing the robot to memorize sketches, they found a way to use the robot's existing knowledge of language to guide its vision. They also added a safety net to make sure the robot doesn't get too confident in its new tricks and forget its original, general knowledge.

The Two-Part Magic Trick

The paper proposes a framework that solves the robot's confusion using two main ideas, which the authors call "text-guided prompting" and "consistency constraints."

1. The Language Translator (Text-Guided Prompting)
Imagine the robot has a library of books about animals. When it sees a photo, it usually just looks at the picture. But with SeCo-SBIR, the researchers tell the robot: "Before you look at the sketch, read the word 'dog' in your library first."

In technical terms, the robot uses its text encoder (the part that understands words) to process a prompt. Instead of just learning new visual patterns from scratch, the robot takes the meaning of the word "dog" from the text side and passes it down to the visual side layer by layer. Think of it like a tour guide (the text) whispering the history of a landmark to a photographer (the visual encoder) before they take a picture. Because the guide already knows the general concept of "dog" from reading millions of books, the photographer doesn't need to memorize every single dog they've ever seen. They just need to understand the idea of a dog. This helps the robot recognize a sketch of a new dog breed it has never seen before, because the "guide" knows what dogs generally look like, regardless of the specific photo.

2. The Strict Teacher (Consistency Constraint)
Now, imagine the robot is learning these new tricks. There's a risk it might get too excited and start hallucinating, thinking every four-legged animal is a "golden retriever" because that's what it saw most often in training. To stop this, the researchers added a "Strict Teacher."

This teacher works by showing the robot two versions of the same photo: one that is slightly messed up (augmented) and one that is clean. The robot's "frozen" brain (the original, untrained version) looks at the messed-up photo, while the "learning" brain looks at the clean one. The teacher demands that both brains agree on what the object is, even though they are looking at different versions of it. If the learning brain starts to drift and say, "This is a cat!" just because it saw a cat in training, the teacher says, "No, look at the original brain; it still sees a dog. You need to stay consistent." This keeps the robot grounded in its original, general knowledge, preventing it from overfitting to the specific training data.

What They Found

The researchers tested this new system on three different "exams" (datasets) that are standard in the field: Sketchy-Ext, TU-Berlin-Ext, and QuickDraw-Ext. These tests involve matching sketches to photos across different categories, including some the robot had never seen before.

The results were quite impressive. On the TU-Berlin-Ext test, which is known for being very tricky because it has many similar-looking categories, SeCo-SBIR improved the accuracy by 5.6% compared to the previous best method. In the "across-dataset" test, where the robot was trained on one set of drawings and tested on a completely different set, it improved accuracy by 6.8%.

The paper suggests that this approach works because it balances two opposing needs: it adapts the robot enough to understand the messy world of sketches, but it keeps the robot's general knowledge intact so it doesn't forget how to handle new categories. The authors found that every part of their system—the language translator, the strict teacher, and the specific math they used to combine these ideas—contributed to the success. When they removed any single part, the robot's performance dropped.

In short, SeCo-SBIR shows that by using language to guide vision and adding a "reality check" to prevent over-learning, we can build AI that is much better at recognizing the world through rough sketches, even when it's seeing those objects for the first time.

Technical Summary: SeCo-SBIR

Problem Statement

Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) aims to retrieve natural images from a database using freehand sketch queries, specifically targeting categories unseen during training. This task faces a fundamental tension: the model must adapt deeply enough to bridge the significant cross-modal gap between abstract sketches and detailed photographs, yet any additional flexibility introduced for adaptation risks overfitting to the seen training categories, thereby eroding the zero-shot generalization capabilities inherent to pre-trained Vision-Language Models (VLMs) like CLIP.

Existing prompt learning methods for ZS-SBIR often struggle with this trade-off. Approaches like CLIP-AT treat text and visual branches in isolation, while bidirectional methods like SpLIP exchange tokens but lack explicit regularization against seen-class overfitting. Consequently, adapted representations often drift toward the distribution of seen categories, failing to generalize to unseen classes, particularly in Generalized ZS-SBIR (GZS-SBIR) settings where seen and unseen classes coexist in the retrieval gallery.

Methodology: SeCo-SBIR

The authors propose SeCo-SBIR (Semantically Consistent Prompt Learning), a framework designed to resolve the adaptation-generalization tension through two complementary mechanisms: text-guided multi-modal prompting and a perturbation-based consistency constraint.

1. Text-Guided Multi-Modal Prompting

Unlike standard visual prompt tuning which learns tokens purely within the visual encoder, SeCo-SBIR routes learnable prompt vectors through the text encoder first.

Mechanism: Learnable prompt vectors replace the initial tokens of a fixed textual template (e.g., "a photo of a [category]"). These vectors pass through the text encoder's transformer layers.
Cross-Modal Coupling: At every layer $l$ of the text encoder, the intermediate prompt representations are projected into the visual encoder's embedding space via a learnable coupling function ( $F_l$ ).
Deep Injection: These projected representations are injected as visual prompts into the corresponding layers of the visual encoder (for both photo and sketch branches).
Rationale: Since the text encoder has learned robust, abstract category-level semantics from large-scale language supervision, this mechanism injects inherently transferable semantic knowledge directly into the visual pathway. This allows the model to adapt to the sketch-photo domain gap while grounding the adaptation in language rather than seen-class visual statistics.

2. Perturbation-Based Consistency Constraint

To prevent the learnable components (prompts and coupling functions) from overfitting to seen categories, the authors introduce a regularization strategy that aligns the adapted model with a frozen CLIP reference.

Asymmetric Design: For a given input, an augmented view (via random erasing, cropping, flipping) is fed to a frozen CLIP visual encoder, while the clean input is fed to the trainable (prompted + adapted) encoder.
InfoNCE Objective: The model is trained using an asymmetric InfoNCE loss. The trainable branch must produce embeddings that are closer to the frozen branch's representation of the augmented input than to representations of other samples in the batch.
Rationale: This prevents trivial solutions (where the trainable model simply copies the frozen model) by forcing the trainable branch to learn input-invariant representations that remain structurally compatible with CLIP's generalizable feature space under perturbation.

3. Training Objective

The framework employs a multi-objective loss function combining:

Cross-Modal Triplet Loss: Pulls matching sketch-photo pairs together and pushes apart non-matching pairs.
NT-Xent Loss: Provides dense batch-level contrastive supervision for both sketch-to-photo and photo-to-sketch directions.
Classification Loss: Aligns visual features with class prototypes derived from the prompt-augmented text encoder.
Consistency Loss: The perturbation-based InfoNCE loss described above.

Key Contributions

Text-Guided Multi-Modal Prompting: A novel mechanism that derives visual prompts from the text encoder's intermediate representations via learnable coupling functions. This leverages pre-existing, language-grounded category semantics to adapt the visual pathway, inherently favoring generalization to unseen classes.
Perturbation-Based Consistency Constraint: A regularization technique using an asymmetric InfoNCE objective to align adapted embeddings with a frozen CLIP reference. This effectively mitigates the risk of overfitting to seen categories introduced by the additional learnable parameters.
State-of-the-Art Performance: The integration of these components with lightweight adapters achieves superior results across all standard ZS-SBIR benchmarks.

Experimental Results

The authors evaluated SeCo-SBIR on three standard benchmarks: Sketchy-Ext, TU-Berlin-Ext, and QuickDraw-Ext, under categorical, generalized (GZS-SBIR), and across-dataset settings.

Categorical ZS-SBIR: SeCo-SBIR achieved the highest scores on all metrics across all datasets. Notably, on TU-Berlin-Ext (sensitive to overfitting due to fine-grained categories), it surpassed the previous best (SpLIP) by +5.6% mAP@all and +6.5% P@100. On QuickDraw-Ext, it improved mAP@all by +1.8%.
Generalized ZS-SBIR (GZS-SBIR): In settings where seen and unseen classes are mixed in the gallery, SeCo-SBIR demonstrated significant gains, improving mAP@all by +4.6% and P@100 by +7.5% on TU-Berlin-Ext compared to SpLIP, indicating effective mitigation of seen-class bias.
Across-Dataset ZS-SBIR: When trained on Sketchy-Ext and tested on TU-Berlin-Ext and QuickDraw-Ext without fine-tuning, the model improved mAP@all by +6.8% on QuickDraw-Ext, highlighting robustness to domain shifts.
Ablation Studies: Experiments confirmed that each component contributes non-redundantly. The text-guided coupling provided a +1.4% gain over independent prompting, and the consistency constraint was critical, preventing a 2.6% performance drop when removed. The asymmetric perturbation design was also validated as superior to using clean inputs for both branches.

Significance and Claims

The paper claims that SeCo-SBIR successfully resolves the core tension in ZS-SBIR: the need for deep adaptation versus the need for zero-shot generalization. By channeling adaptation through language-grounded knowledge (via the text encoder) and explicitly constraining learnable components (via the consistency constraint), the framework adapts the model to the sketch-photo domain without sacrificing the generalization capabilities of the pre-trained CLIP backbone.

The authors position this work as a principled approach for deploying vision-language models in scenarios demanding generalization to unseen categories, demonstrating that semantic consistency and perturbation-based regularization are effective strategies for overcoming the limitations of current prompt learning paradigms in cross-modal retrieval.