💻 computer science

Locality-Preserving Sketch Token Fusion for High-Fidelity Single-View 3D Reconstruction

This paper proposes a novel framework called Locality-Preserving Sketch Token Fusion that achieves high-fidelity single-view 3D reconstruction by maintaining local sketch details through a dual-path representation and a cross-attention mesh sampler, effectively overcoming the geometric detail loss inherent in existing global feature compression methods.

Original authors: Lei Qinchuan, Li Nannan, Zhong Jiaming, Luo XianZeng, Zhao Min

Published 2026-10-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lei Qinchuan, Li Nannan, Zhong Jiaming, Luo XianZeng, Zhao Min

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Creating a three-dimensional object from a single two-dimensional drawing is a task that feels almost magical to the human eye, yet it remains one of the most stubborn puzzles in computer science. For decades, artists and engineers have relied on complex software and years of training to build digital models, but the dream has always been to let anyone simply sketch a shape on a piece of paper and have a computer instantly understand its full volume. The difficulty lies in the nature of a sketch itself: a hand-drawn line is ambiguous. It lacks the depth, lighting, and precise boundaries of a photograph, and it often omits parts of an object that are hidden from view. When a computer tries to turn these sparse lines into a solid 3D shape, it often struggles to remember the small, delicate details, such as the thin tail of an airplane or the intricate curve of a chair back, because the standard methods used to process these images tend to blur them away in favor of the big picture.

A team of researchers from Macau and Guangdong has proposed a new way to solve this problem, one that changes how computers "read" a sketch. Instead of forcing the computer to squash the entire drawing into a single, simplified summary of the object, their method keeps the local details alive throughout the process. They developed a system that treats a sketch not as a flat image to be flattened further, but as a collection of distinct, spatially organized pieces of information. By preserving the specific location and shape of every stroke, the computer can refer back to the original drawing while it builds the 3D model, ensuring that fine features are not lost in translation. This approach allows the system to generate high-quality 3D meshes that are far more faithful to the original hand-drawn intent, capturing thin structures and complex local geometries that previous methods often smoothed over or distorted.

The core of this new framework is a shift in how the computer processes the visual data. Traditionally, when a computer analyzes an image to understand a shape, it often uses a technique that aggregates all the visual information into one compact vector, a single mathematical representation that captures the general idea of the object. While this is efficient for recognizing that an image contains a chair or a car, it inevitably discards the specific spatial relationships between different parts of the drawing. The researchers found that this loss of local information is exactly what causes the computer to fail at reconstructing fine details. To fix this, they introduced a dual-path system. One path still learns the overall shape and the angle from which the object is viewed, but the second path maintains a structured set of tokens, or small data packets, that correspond to specific regions of the sketch. These tokens act as a map, keeping the local geometric patterns of the strokes intact, such as the sharp angle of a corner or the curve of a handle, separate from the general shape summary.

Once the computer has this dual representation, it must figure out how to use these local clues to deform a basic 3D template into the final shape. The researchers designed a mechanism that allows the global understanding of the object to actively query the local tokens for help. Imagine the computer holding a 3D model of a generic chair and asking, "Where exactly should the backrest curve?" It then looks at the specific tokens in the sketch that correspond to the backrest area to retrieve the precise geometric instructions needed for that part. This process, which the team calls a local token cross-attention module, creates a dynamic link between the 2D drawing and the 3D model. Instead of guessing the details based on a vague memory of the whole image, the system retrieves the exact evidence from the drawing at the moment it needs to define a specific vertex on the 3D mesh. This ensures that the final model aligns perfectly with the original strokes, preserving the thin structures and local nuances that define the object's character.

To test their idea, the researchers trained their system on a large dataset of synthetic 3D models and their corresponding sketches, covering thirteen different categories of objects ranging from airplanes and cars to lamps and sofas. They then evaluated the system's performance against existing methods using standard benchmarks. The results showed a clear improvement in accuracy. When measuring how closely the generated 3D shapes matched the true ground-truth models, their method achieved higher scores across almost every category. For instance, in the case of airplanes, the system was able to reconstruct the delicate tail structures with much greater fidelity than previous approaches, which often produced distorted or missing tails. Similarly, for chairs and benches, the new method successfully recovered the specific topology and backrest shapes that other systems had flattened or misshapen. The team also tested the system on a dataset of real, hand-drawn sketches created by humans, which are often messier and more ambiguous than computer-generated ones. Even with these challenging inputs, the system maintained its ability to produce geometrically plausible and detailed 3D shapes, demonstrating that the method is robust enough to handle the variations found in real-world drawing.

The success of this approach suggests that the key to better 3D reconstruction from sketches is not just in understanding the object as a whole, but in respecting the local information contained within the drawing. By refusing to compress the sketch into a single, oversimplified summary, the researchers have shown that it is possible to retain the fine-grained details that make a model look real. The system does not just guess the shape; it actively consults the drawing's local features to guide the construction of the 3D mesh. This capability is particularly important for objects with thin components or complex surface variations, where a loss of detail can make the difference between a recognizable object and a distorted blob. The researchers acknowledge that while their method represents a significant step forward, there is still room for improvement, particularly in handling the most ambiguous sketches or objects with extremely complex geometries. However, the findings provide a strong foundation for future work, pointing toward a future where the barrier between a simple sketch and a detailed 3D model is significantly lowered, making digital creation more accessible to everyone.

Technical Summary: Locality-Preserving Sketch Token Fusion for High-Fidelity Single-View 3D Reconstruction

1. Problem Statement

Reconstructing high-fidelity 3D shapes from a single freehand sketch is a challenging task due to the inherent ambiguity of hand-drawn strokes and the loss of fine-grained geometric details during feature encoding. While existing deep learning approaches have advanced sketch-based 3D modeling, they typically rely on an encoder-decoder paradigm that compresses 2D sketch features into compact global latent representations via flattening or pooling operations.

This compression strategy, while effective for capturing coarse shape semantics and viewpoint information, inevitably discards spatially localized stroke cues. Consequently, the spatial correspondence between image features and local sketch structures is weakened, leading to the loss of critical geometric details such as thin protrusions, local concavities, and detailed contours. During decoding, mesh generation often occurs via a single feed-forward process regressing vertex deformation from a global vector, lacking an explicit mechanism to adaptively access local sketch evidence. This results in reconstructed models that preserve overall shape but suffer from over-smoothed local regions and fail to fully capture the geometric intent of the original sketch.

2. Methodology

The authors propose Locality-Preserving Sketch Token Fusion (LP-STF), a framework designed to maintain and exploit local sketch information while preserving global shape understanding. The architecture consists of five primary components:

2.1 Feature Extractor and Token Transformer Block (TTB)

Instead of aggregating features into a single vector, the input sketch is processed by a lightweight convolutional encoder to produce a spatial feature map. This map is transformed into a fixed spatial resolution ( $7 \times 7$ ) to generate local sketch tokens.

Dual-Path Representation: The framework introduces a Token Transformer Block that transforms these features into complementary global and local representations.
Mechanism: The feature map is flattened into a token sequence and processed through multi-head self-attention and an MLP to capture long-range structural relationships.
Gated Fusion: A learnable gated residual fusion mechanism ( $F_{token} = X_t + \tan(\alpha) \cdot X_r$ ) adaptively balances the original sketch information with the transformed features. This preserves fine-grained stroke patterns while enhancing discriminative geometric patterns, resulting in a locality-preserving token representation.

2.2 Encoder-Decoder Module

The locality-preserving tokens are factorized into two complementary latent codes:

Shape Representation ( $F_s$ ): Encodes intrinsic geometric characteristics less sensitive to viewpoint variations.
Viewpoint Representation ( $F_v$ ): Explicitly encodes the viewing configuration of the input sketch.
Disentanglement: A contrastive alignment objective ( $L_{con}$ ) is imposed to ensure $F_v$ accurately captures viewpoint information by aligning it with ground-truth view embeddings while distinguishing it from other candidates. The decoder aggregates these into a global shape latent ( $Z_{global}$ ).

2.3 Local Token Cross-Attention (LTCA) Module

To bridge the gap between the global latent representation and local geometric details, the authors design a Local Token Cross-Attention Module.

Query-Driven Sampling: The global shape latent ( $Z_{global}$ ) serves as the query ( $Q$ ), while the locality-preserving sketch tokens serve as the key ( $K$ ) and value ( $V$ ).
Adaptive Retrieval: Through scaled dot-product attention, the global representation adaptively queries and retrieves relevant local sketch tokens ( $Z_{local}$ ). This allows the network to dynamically identify sketch regions pertinent to the current geometric reconstruction (e.g., object boundaries or thin structures).
Fusion: The retrieved local cues are fused with the global representation via residual addition ( $Z_f = Z_{global} + Z_{local}$ ), creating a geometry-aware representation that combines global semantics with fine-grained local evidence.

2.4 Mesh Generator and Training

The fused representation is decoded by a Mesh Generator to produce an initial mesh, which is further refined by a mesh refinement module. The framework is trained using three complementary objectives:

Silhouette Reconstruction ( $L_s$ ): A differentiable Intersection-over-Union (IoU) loss between the rendered silhouette of the predicted mesh and the target silhouette extracted from the sketch.
Mesh Regularization ( $L_r$ ): Includes Laplacian smoothing ( $L_{lap}$ ) and edge length regularization ( $L_{edge}$ ) to ensure geometric plausibility and surface smoothness.
View-Aware Contrastive Alignment ( $L_{con}$ ): Ensures the viewpoint branch accurately disentangles view-dependent cues from intrinsic shape information.

3. Key Contributions

The paper outlines three primary contributions:

Token Transformer Block for Locality-Preserving Dual-Path Representation: A novel encoding strategy that avoids excessive global compression by jointly learning global shape semantics and spatially structured local tokens, preserving fine-grained stroke information.
Local Token Cross-Attention Module for Geometry-Aware Alignment: An adaptive mechanism that enables the decoder to selectively retrieve informative local sketch regions during mesh generation, establishing a more precise correspondence between 2D stroke structures and 3D vertex deformations.
High-Fidelity and Robust Reconstruction: The framework achieves superior performance on both synthetic and real hand-drawn sketches, particularly improving the reconstruction of objects with thin structures and complex local geometries.

4. Experimental Results

The method was evaluated on the ShapeNet-Synthetic and ShapeNet-Sketch datasets across 13 object categories (e.g., airplanes, chairs, cars).

Quantitative Performance:
- On ShapeNet-Synthetic, the proposed method achieved the highest average IoU scores under both ground-truth viewpoint (0.628) and predicted viewpoint (0.610) settings, outperforming state-of-the-art baselines like SS2M, Sketch2Model, and Magic3DSketch.
- On ShapeNet-Sketch (real hand-drawn sketches), the method demonstrated strong generalization, achieving the highest average IoU (0.498 with GT view, 0.487 with predicted view) across the dataset.
Qualitative Analysis:
- Visual comparisons show that LP-STF successfully recovers fine-grained details often lost in baseline methods, such as airplane tails, chair backrests, square bench topologies, and car rear-wings.
- Ablation studies confirm that removing either the Token Transformer Block (TTB) or the Local Token Cross-Attention (LTCA) module leads to a measurable drop in IoU and a visible loss of structural fidelity in thin or complex regions.

5. Significance and Claims

The paper claims that the primary significance of this work lies in addressing the fundamental limitation of existing encoder-decoder paradigms: the loss of spatially localized stroke information due to global feature compression.

By introducing a dual-path representation and an adaptive cross-attention mechanism, the framework successfully maintains the spatial correspondence between 2D sketch structures and 3D geometry. The authors assert that this approach is particularly beneficial for recovering "thin structures, elongated components, and distinctive local geometric patterns" that are difficult to infer from limited 2D inputs. The results suggest that explicitly preserving and exploiting local sketch tokens is critical for achieving faithful, high-fidelity 3D reconstruction from single-view sketches, bridging the gap between professional design requirements and accessible, non-expert sketching interfaces.

Future work, as noted by the authors, will focus on exploring more effective hierarchical token representations and incorporating additional cues (such as depth or semantic information) to further constrain the reconstruction of geometrically complex objects.