🤖 AI

Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval

This paper proposes the STBIR framework, which synergizes hand-drawn sketches and textual descriptions through curriculum learning, category-knowledge optimization, and multi-stage cross-modal alignment to achieve superior fine-grained image retrieval performance, supported by a newly curated benchmark dataset.

Original authors: Siyuan Wang, Hanchen Gao, Guangming Zhu, Jiang Lu, Yiyue Ma, Tianci Wu, Jincai Huang, Liang Zhang

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyuan Wang, Hanchen Gao, Guangming Zhu, Jiang Lu, Yiyue Ma, Tianci Wu, Jincai Huang, Liang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific pair of shoes in a massive, chaotic warehouse containing millions of pairs. You have two ways to describe what you are looking for:

The Sketch: You draw a quick picture of the shoe. It's great at showing the shape (is it a sneaker or a boot? Does it have a high heel?), but it's just black lines on white paper. It has no color or texture.
The Text: You write a description like "red leather running shoe with white stripes." This is great for color and material, but it's terrible at describing complex shapes or weird designs.

The Problem:
Previous computer systems were like a detective who only listened to either the drawing or the description, but never both at the same time. If you only showed the drawing, the computer couldn't tell the difference between a red shoe and a blue shoe. If you only used the text, it might pick the right color but the wrong shape. They also struggled when the drawings were messy or the descriptions were vague.

The Solution: The "STBIR" Team-Up
The researchers in this paper built a new system called STBIR (Sketch and Text Based Image Retrieval). Think of STBIR as a super-smart detective who is trained to combine the "shape" of the sketch with the "details" of the text to find the perfect match.

Here is how they made this detective so good, using three simple tricks:

1. The "Training with Obstacles" Trick (Curriculum Learning)

Imagine teaching a child to ride a bike. You don't start them on a steep, rocky hill with a strong wind. You start on a flat, smooth driveway.

How STBIR does it: The system starts training on perfect, clean sketches and clear descriptions. As it gets better, the researchers intentionally add "noise" (like making the sketch messy or the text vague) to the training data.
The Result: Just like the child learns to handle wind and rocks, the computer learns to find the right shoe even if your sketch is scribbled or your description is a bit fuzzy. It becomes robust.

2. The "Grouping by Category" Trick (Feature Space Optimization)

Imagine a library where all the books are thrown into one giant pile. Finding a specific book is hard. Now, imagine a librarian who organizes books not just by title, but by genre and author, creating tight little groups for similar items.

How STBIR does it: The system uses "category knowledge" (like knowing the difference between a "sneaker" and a "slipper") to squeeze similar items closer together in its memory and push different items further apart.
The Result: The computer becomes much better at telling the difference between two very similar shoes, rather than getting confused.

3. The "Three-Step Dance" Trick (Multi-Stage Alignment)

This is the most clever part. Imagine trying to get three people (a Sketcher, a Writer, and a Photographer) to agree on what a "red shoe" looks like. If you ask them all to shout their ideas at the same time, it's a chaotic mess, and no one listens.

How STBIR does it: They teach them in a specific order:
1. Step 1: First, they teach the Sketcher to understand the Photographer's language (since shapes are similar).
2. Step 2: Next, they teach the Photographer to adjust their view based on the Sketcher's lines.
3. Step 3: Finally, they bring in the Writer to add the color and texture details to the now-agreed-upon shape.
The Result: By taking it one step at a time, the three different types of information (drawing, words, photo) line up perfectly without getting confused or fighting each other.

The New "Test Book" (The Dataset)

To prove their system works, the researchers realized there was no good "test book" available. Existing tests either had fake drawings or didn't have enough variety.
So, they created the STBIR Dataset.

They took real photos of shoes, chairs, and daily objects.
They hired people to draw real sketches of them.
They used a smart AI to write detailed descriptions for them.
They checked everything manually to make sure the drawing, the words, and the photo all matched perfectly.

The Verdict

When they tested their new "super detective" (STBIR) against all the old methods, it won every time. It was much better at finding the exact item you wanted, even when the clues were imperfect.

In short: This paper teaches computers how to look at a messy drawing and a short description simultaneously, learn to ignore the mistakes, and find the exact object you are looking for in a sea of millions of options.

1. Problem Statement

Fine-grained image retrieval (FG-IR) aims to retrieve specific instances from a large database based on user queries. The paper addresses the limitations of using a single modality for this task:

Hand-drawn Sketches: Excellent at capturing geometric contours and structural topology but lack color, texture, and material information.
Textual Descriptions: Effective at describing attributes like color, texture, and material but struggle to convey complex spatial structures or irregular shapes.
The Challenge: Relying on only one modality leads to suboptimal retrieval, especially in fine-grained scenarios where instances differ by non-geometric attributes (e.g., color). Furthermore, existing multimodal approaches suffer from:
- Modality Gaps: Significant distribution mismatches between sketch, text, and image feature spaces.
- Optimization Instability: Gradient imbalances and parameter divergence during joint training of multimodal encoders.
- Data Scarcity: A lack of high-quality, publicly available datasets containing aligned triplets of hand-drawn sketches, text, and natural images.

2. Methodology: The STBIR Framework

The authors propose the Sketch and Text Based Image Retrieval (STBIR) framework, which synergizes structural contours from sketches with descriptive attributes from text. The framework consists of three core modules and a novel training strategy:

A. Feature Encoders

Sketch Encoder: Uses ResNet to extract global features from hand-drawn sketches, chosen for its ability to align well with image modalities.
Text & Image Encoders: Utilizes CLIP (Contrastive Language-Image Pre-training) to leverage its strong pre-trained alignment between visual and textual semantics.

B. Core Modules

Curriculum Learning-Driven Robustness Enhancement (CLDRE):
- Goal: Improve model robustness against low-quality or noisy queries.
- Mechanism: Simulates cognitive evolution by progressively injecting noise into input features during training. Early stages use high-fidelity data; as training progresses ( $t$ increases), noise intensity ( $\alpha$ ) increases. This forces the model to learn stable mappings despite degraded inputs.
Category-Knowledge-Based Feature Space Optimization (CKFSO):
- Goal: Enhance instance-level discriminability.
- Mechanism: Integrates category priors as constraints using Additive Angular Margin Loss (ArcFace). This compacts intra-class distributions (making same-category items closer) while expanding inter-class margins, significantly boosting feature representation quality.
Multi-Stage Cross-Modal Feature Alignment (MCFA):
- Goal: Mitigate gradient imbalance and parameter divergence caused by the modality gap.
- Mechanism: Instead of updating all encoders simultaneously, the training is decoupled into three sequential stages:
  1. Sketch Mapping: Freeze CLIP (Image/Text); optimize only the Sketch encoder to map sketch features into the pre-aligned Image-Text space.
  2. Image Refinement: Freeze Sketch/Text; fine-tune the Image encoder to align with the sketch's geometric priors.
  3. Text Integration: Freeze Sketch/Image; fine-tune the Text encoder to embed fine-grained attributes (color/texture) into the established joint space.
- Loss Function: The total loss combines Additive Angular Margin Loss ( $L_{AAML}$ ), InfoNCE loss, and Triplet loss.

3. Key Contributions

STBIR Framework: A novel architecture that effectively fuses sketch structural data with text attribute data, outperforming state-of-the-art methods.
Novel Training Mechanisms:
- Introduction of CLDRE for robustness against query variations.
- Introduction of CKFSO for enhanced discriminability via category knowledge.
- Proposal of MCFA, a multi-stage alignment strategy that prevents feature space perturbations and ensures stable convergence.
STBIR Dataset: The creation and release of a comprehensive tri-modal benchmark dataset comprising three subsets:
- STBIR-S: Single-category (Shoes), 13 subcategories.
- STBIR-C: Single-category (Chairs), 24 subcategories.
- STBIR-D: Large-scale daily objects, 125 categories.
- Significance: It is the first public dataset with strictly aligned, hand-drawn sketches, text, and images at the fine-grained instance level.

4. Experimental Results

The framework was evaluated on the STBIR-S, STBIR-C, and STBIR-D datasets against baselines like CLIP, TASKformer, Pic2Word, and SEARLE.

Performance: STBIR achieved State-of-the-Art (SOTA) results across all datasets.
- On STBIR-C, it achieved an R@1 of 57.88, significantly outperforming the second-best model (Pic2Word/TASKformer at ~53.38).
- On STBIR-D (large scale), it secured top ranks in R@1 (62.85) and R@5 (93.44), demonstrating robustness in complex, multi-category scenarios.
Ablation Studies:
- Removing any single modality (Text or Sketch) caused a drastic drop in performance, confirming the necessity of synergy.
- Replacing the Multi-Stage (MCFA) strategy with synchronous updates led to a sharp decline in R@1, proving the importance of sequential alignment.
- The S → I → T (Sketch $\to$ Image $\to$ Text) training order yielded the best results, validating the hypothesis that establishing a visual-geometric baseline before introducing abstract semantics is optimal.

5. Significance

Theoretical Advancement: The paper provides a robust solution to the "modality gap" problem in fine-grained retrieval by decoupling the optimization of different modalities, addressing the instability often found in joint multimodal learning.
Practical Impact: The proposed framework enables more intuitive and accurate product search (e.g., e-commerce) where users can combine rough sketches with specific text descriptions.
Community Resource: The release of the STBIR dataset fills a critical gap in the research community, providing a standardized benchmark for tri-modal fine-grained retrieval, which was previously hindered by the lack of high-quality, human-drawn sketch-text-image triplets.

In conclusion, STBIR demonstrates that by strategically fusing the structural strengths of sketches with the descriptive power of text, and by carefully managing the training dynamics through curriculum learning and staged alignment, it is possible to achieve superior fine-grained image retrieval performance.