💻 computer science

Category-Level Fine-Grained Sketch-Based 3D Shape Retrieval

This paper addresses the challenge of category-level fine-grained sketch-based 3D shape retrieval by introducing a new dataset with 54 sub-categories and proposing a method featuring specialized feature extractors and optimized cross-domain alignment to effectively retrieve 3D shapes matching the fine-grained details of input sketches.

Original authors: Bingrui Wang, Yuan Zhou, Sun-Yuan Kung

Published 2026-08-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingrui Wang, Yuan Zhou, Sun-Yuan Kung

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, three-dimensional shapes are the building blocks of our virtual worlds, powering everything from video game characters to architectural models and industrial designs. As the volume of these digital objects grows into the millions, finding a specific one quickly becomes a daunting task. While people can search for these objects using text keywords or by uploading a photo of a similar item, there is a more intuitive way to search: drawing. A hand-drawn sketch captures the essence of an object—the overall structure and key details—without needing perfect realism. This method of searching for 3D objects using a simple 2D drawing is known as sketch-based 3D shape retrieval. However, current systems are often too broad for practical use. They might successfully find a "chair" when asked, but they struggle to distinguish between a specific type of chair, such as a "cantilever" chair versus a "ladder-back" chair. This lack of precision limits the technology's usefulness in real-world scenarios where users need to find exact matches, not just general categories.

A team of researchers from Tianjin University and Princeton University has taken a significant step toward solving this problem by tackling the challenge of fine-grained retrieval. Their work addresses the difficulty of finding 3D shapes that belong to the same specific sub-category as a user's sketch, rather than just the same broad category. To do this, they first had to create a new foundation, as no suitable data existed for this specific task. They assembled a massive new dataset containing 7,666 hand-drawn sketches and 20,445 corresponding 3D shapes. These items were carefully organized into 54 distinct sub-categories across three broad groups: airplanes, cars, and chairs. For instance, within the "car" category, they did not just group all vehicles together; they separated them into specific types like sedans, trucks, and buses. To ensure the data was accurate and balanced, they enlisted domain experts to define these sub-categories and supervised graduate students to re-classify existing images and 3D models, while also having students draw new sketches to fill any gaps in the collection.

With this new dataset in hand, the researchers developed a specialized system to bridge the gap between a flat 2D drawing and a complex 3D object. They recognized that a standard computer vision tool, designed to recognize photos of real-world objects, would not work well for hand-drawn sketches, which often contain large areas of empty space and vary wildly in style. Consequently, they designed a new type of feature extractor specifically for sketches, adjusting the network's structure to better handle the unique characteristics of human drawing. For the 3D shapes, they created a system that looks at the object from multiple angles, much like a person walking around a sculpture to see it from every side. This system pays special attention to small, critical details on the object's surface, identifying and scoring the most important parts of the shape to create a precise digital fingerprint.

The most innovative part of their approach involves how these two different types of data are taught to understand each other. Since 3D models are generally more detailed and easier for computers to analyze than rough sketches, the researchers used the 3D models as a guide. They first trained the system to understand the 3D shapes perfectly. Then, they used that understanding to teach the system how to interpret the sketches, effectively using the richer information from the 3D world to compensate for the abstraction and scarcity of the drawing data. This "3D shape-guided" learning strategy allowed the system to align the two different types of data in a shared space, making it possible to match a sketch to its exact 3D counterpart even when the visual differences between them are vast.

When the researchers tested their new method, the results were striking. On their new fine-grained dataset, their system significantly outperformed all existing methods, improving the accuracy of finding the correct sub-category by more than double in some cases. It successfully retrieved the right airplane, car, or chair sub-type far more often than previous technologies could. Furthermore, the system proved to be robust; when tested on older, broader datasets that were not designed for fine-grained tasks, it still performed better than the current state-of-the-art methods. This suggests that the techniques they developed are not just a narrow fix for a specific problem but a more general improvement in how computers understand the relationship between drawings and 3D objects. By creating a dedicated dataset and a tailored learning strategy, this work moves the field closer to a future where a quick sketch can reliably find the exact digital object a user needs.

Technical Summary: Category-Level Fine-Grained Sketch-Based 3D Shape Retrieval

Problem Definition

The paper addresses Category-Level Fine-Grained Sketch-Based 3D Shape Retrieval (FG-SBSR). Unlike traditional SBSR, which retrieves 3D shapes based on broad categories (e.g., "chair"), FG-SBSR aims to retrieve shapes belonging to specific sub-categories (e.g., "cantilever chair" vs. "ladder-back chair") based on a user's free-hand sketch.

The task faces three primary challenges:

Abstraction and Cross-Domain Gap: Sketches are highly abstract 2D representations, while 3D shapes are volumetric. Bridging the semantic and structural gap between 2D sketches and 3D objects is inherently difficult.
Fine-Grained Similarity: Sub-categories within a broad class often share high visual similarity, requiring the model to capture subtle intra-category structural differences rather than just inter-category separation.
Data Scarcity: Existing SBSR datasets are coarse-grained. There is a lack of paired sketch-3D shape data annotated at the sub-category level, creating a significant bottleneck for training deep learning models in this specific domain.

Methodology

The authors propose a framework comprising a new dataset, specialized feature extractors, and a novel training strategy.

1. The FG-SBSR Dataset

To address the data bottleneck, the authors constructed a new dataset containing:

7,666 sketches and 20,445 3D shapes.
54 fine-grained sub-categories across three broad classes: Airplanes, Cars, and Chairs.
Construction Process: Graduate students, supervised by domain experts, re-classified samples from existing repositories (Tu-Berlin, Sketchy, Pro-Chair-3D, FG3D) and manually drew new sketches to ensure balanced representation (minimum 12 samples per sub-category).

2. Feature Extraction Architecture

The framework employs two distinct feature extractors optimized for the specific characteristics of sketches and 3D shapes:

Fine-Grained 3D Shape Feature Extractor:
- Utilizes a multi-view approach where 3D shapes are rendered into 2D views.
- Introduces a Key Parts Extractor (KPE): A deep network module that generates anchor points on feature maps, delineates regions, and scores them to identify discriminative local details.
- Fusion Strategy: The top- $K$ scoring part features are fused via a partial attention vector. These are concatenated with the spatial attention (SA) fine-tuned overall view feature. Finally, a channel-level attention vector aggregates features across all views to produce the final 3D shape representation.
Fine-Grained Sketch Feature Extractor:
- Designed specifically for the high abstraction and stylistic variance of human sketches, rather than using standard photo-oriented CNNs (like ResNet-50) directly.
- Modifications: Increases convolution kernel sizes in the first two layers, reduces network depth, and applies higher dropout rates in the final fully connected layers to handle the sparse and abstract nature of sketch data.

3. 3D Shape-Guided Sketch Learning

Recognizing that 3D shape data is more detailed, regular, and abundant than sketch data, the authors propose a two-stage training strategy to bridge the domain gap:

Stage 1: Train the 3D shape feature extractor using prototype contrastive loss.
Stage 2: Use the pre-trained 3D shape extractor to guide the training of the sketch feature extractor.
- The sketch extractor is trained to align its features with the 3D shape prototypes in a shared semantic space.
- This approach leverages the richer priors of 3D data to compensate for the high abstraction and scarcity of sketch data.

Key Contributions

Task Definition and Dataset: The paper formally defines the category-level FG-SBSR task and contributes the first dataset specifically designed for this scenario, containing 54 fine-grained sub-categories.
Novel Methodology:
- A cross-view part-aware 3D extractor that explicitly mines and fuses discriminative local details from multi-view projections.
- A 3D shape-guided sketch learning strategy that utilizes 3D data priors to overcome sketch abstraction and data scarcity.
Performance Validation: The proposed method sets a new state-of-the-art on the new FG-SBSR benchmark and demonstrates robust generalization to existing coarse-grained SBSR benchmarks (SHREC'14 and D2O).

Experimental Results

Extensive experiments were conducted on the new FG-SBSR dataset (Airplane, Car, Chair subsets) and two coarse-grained benchmarks.

FG-SBSR Performance: The proposed method significantly outperformed existing state-of-the-art methods (including Siamese, DCHML, TCL, CGN, JFLN, and PLSB).
- On the Airplane subset, the Nearest Neighbor (NN) metric improved from 0.161 (achieved by the JFLN baseline) to 0.369, representing an improvement of over 100% as noted in the paper.
- Similar substantial gains were observed for Cars and Chairs across all metrics (NN, FT, ST, nDCG, MRR, mAP).
Generalization: When tested on coarse-grained benchmarks (SHREC'14-SUB and D2O), the method maintained superior performance, indicating it does not overfit to the fine-grained setting.
Ablation Studies:
- Removing the fine-grained 3D extractor (FG3D) or the fine-grained sketch extractor (FGSK) resulted in significant performance drops.
- Removing the 3D-guided alignment (GA) also degraded performance, confirming the efficacy of using 3D data to guide sketch learning.
Intra-Domain Analysis: Experiments showed that 3D-to-3D retrieval performance is significantly higher than Sketch-to-Sketch retrieval, validating the hypothesis that 3D shapes are more feature-centralized and learnable, thus justifying the guided learning approach.

Significance and Claims

The paper claims that this work represents a "notable step" for Sketch-Based 3D Shape Retrieval (SBSR) toward real-world applications. By moving from coarse-grained to fine-grained scenarios, the research addresses the practical difficulty users face when needing specific 3D models rather than general categories. The authors posit that the proposed dataset and methodology provide the necessary foundation for advancing SBSR in scenarios where subtle structural differences are critical. The work is presented as a solution to the data scarcity and domain gap challenges that have previously hindered fine-grained SBSR research.