Order Matters: 3D Shape Generation from Sequential VR Sketches

1. Problem Statement

The paper addresses the challenge of generating high-fidelity 3D shapes from sequential Virtual Reality (VR) sketches. While VR sketching offers an intuitive, immersive alternative to traditional CAD for rapid ideation, existing "sketch-to-shape" models suffer from three critical limitations:

• Data Scarcity: There is a lack of large-scale, paired datasets containing VR sketches and corresponding 3D meshes. Existing benchmarks (e.g., 3DVRChair) are limited to single categories and small sample sizes.
• Geometric Misalignment: Human-drawn VR sketches often contain depth perception errors and spatial inaccuracies, making it difficult to align them with target 3D shapes for training and evaluation.
• Temporal Information Loss: Current approaches treat VR sketches as unordered point clouds, discarding the temporal order of strokes and points. The authors argue that stroke order encodes crucial information about connectivity, structure, and design intent, which is essential for accurate reconstruction.

2. Methodology

The authors propose VRSKETCH2SHAPE, a framework comprising a novel dataset, a synthetic data generation pipeline, and an order-aware deep learning model.

A. Dataset and Data Generation

To overcome data scarcity and alignment issues, the authors created the VRSKETCH2SHAPE dataset:

• Real Data Collection: They developed a custom Unity-based VR interface with a surface-snapping mechanism. This tool projects user-drawn points onto the underlying 3D reference model, ensuring geometric alignment and reducing annotation noise. They collected 900 hand-drawn sketches across four categories (chairs, tables, cabinets, airplanes).
• Synthetic Data Generation: To scale training, they introduced a learning-free, heuristic pipeline to generate sequential VR sketches from arbitrary 3D meshes. The pipeline:
  1. Samples salient points on the mesh surface (using Sharp Edge Sampling).
  2. Fits Bézier splines to these points to form candidate strokes.
  3. Merges and simplifies strokes.
  4. Orders strokes and points using a depth-first traversal (DFS) on a connectivity graph to approximate human drawing order.
  • This pipeline generated 20,838 synthetic sketch-shape pairs.

B. Model Architecture

The core model consists of an Order-Aware Sketch Encoder coupled with a Diffusion-Based 3D Generator.

1. Sketch Encoder (Modified BERT):

   • Tokenization: A sketch is tokenized as a sequence of 3D points, with special tokens (SEP for stroke end, EOS for sketch end) separating strokes.
   • Embeddings:
     • Spatial: 3D coordinates are encoded using Fourier features to capture high-frequency geometric details.
     • Sequential: Stroke indices and point indices are encoded using 1D sinusoidal Fourier features.
   • Processing: The embeddings are summed and passed through a lightweight BERT-style transformer to capture temporal dependencies between strokes and points.
   • Augmentation: During training, the model uses stroke dropping, point dropping, and stroke swapping to improve robustness.

2. Shape Generator (SDFUSION):

   • The encoded sketch tokens serve as conditioning inputs for SDFUSION, a latent diffusion model.
   • The model operates in the latent space of a pretrained 3D VQ-VAE (which encodes Signed Distance Functions).
   • A U-Net predicts the denoised latent representation, which is then decoded into a 3D mesh.
   • The system is trained end-to-end, jointly optimizing the encoder and the diffusion model.

3. Key Contributions

1. VRSKETCH2SHAPE Dataset: The first open-access, multi-category dataset containing both 20k+ synthetic and 900 real sequential VR sketch–shape pairs. It is the first to preserve full stroke and point ordering.
2. Automatic Synthetic Pipeline: A novel, training-free heuristic pipeline that generates realistic sequential sketches from 3D meshes, enabling large-scale supervised learning without manual annotation.
3. Order-Aware Model: A novel architecture that explicitly models the temporal sequence of strokes using a transformer-based encoder, demonstrating that stroke order is a critical signal for 3D reconstruction.
4. Robust Generalization: The model achieves strong performance on real-world sketches even when trained solely on synthetic data (zero-shot), and further improves with minimal fine-tuning.

4. Results

• Quantitative Performance: On the 3DVRChair benchmark and their own VRSKETCH2SHAPE test set, the proposed method significantly outperforms state-of-the-art baselines (e.g., Luo et al. [42], LAS-Diffusion).
  • Chamfer Distance (CD): Reduced by >60% compared to previous methods.
  • F-score: Improved by >40%.
• Generalization: The model trained on synthetic data alone generalizes effectively to real sketches. With as few as 50 real sketches per category for fine-tuning, it reaches near-optimal performance.
• Partial Sketch Completion: The model demonstrates strong "cross-modal shape completion" capabilities, successfully inferring complete 3D shapes from partial sketches (e.g., only the first 50% of strokes), outperforming point-cloud-based baselines.
• Free-Hand Sketching: The model generalizes well to free-hand sketches drawn without reference shapes, producing structurally accurate and detailed geometries.

5. Significance

This work fundamentally shifts the paradigm of sketch-based 3D generation by proving that temporal order matters. By treating VR sketches as ordered sequences rather than unordered point clouds, the authors unlock structural cues that were previously ignored.

• Practical Impact: The framework enables rapid, intuitive 3D design workflows where users can sketch in VR and receive immediate, high-fidelity 3D prototypes.
• Data Efficiency: The success of the synthetic-to-real transfer suggests that large-scale, high-quality 3D generative models can be trained without the prohibitive cost of collecting massive amounts of human-drawn data.
• Open Science: The release of the dataset, code, and models lowers the barrier for future research in VR-based design and 3D generative AI.