Technical Summary: Img2CADSeq

Problem Statement

While recent advancements in single-view 3D generation (e.g., NeRF, Gaussian Splatting) have achieved remarkable visual fidelity, the resulting representations are typically implicit fields or unstructured point splats. These formats lack the explicit internal topology of geometric curves and surfaces required for precision engineering tasks. Consequently, they are ill-suited for industrial design and manufacturing, which rely on Boundary Representations (BReps) in standard formats like STEP. Reconstructing high-quality BReps from single images remains a significant challenge due to the complexity of topological constraints, the unstructured nature of long operation sequences, and the scarcity of industrial-grade training data that bridges the gap between synthetic renders and real-world manufacturing artifacts.

Methodology

Img2CADSeq is a multi-stage pipeline designed to generate standardized STEP files from single-view images (or point clouds) by encoding CAD operations into a discrete latent space. The framework consists of three primary stages:

1. Hierarchical Sequence Encoding

To address the difficulty of learning long, unstructured CAD sequences, the authors propose a three-level hierarchical codebook inspired by the "profile-to-detail" design process. This strategy quantizes CAD operations into three distinct levels using Vector Quantized Variational Autoencoders (VQ-VAE):

• Extrude-Block (EB): Encodes global semantics, including sketch plane orientation, origin, extrusion depth, and Boolean operation types (New, Join, Cut).
• Sketch-Patch (SP): Encodes topological layout. Unlike simple sorting, this layer uses a scoring mechanism prioritizing dominant profiles (based on area and diagonal scale) to establish spatial anchors for subsequent loops.
• Curve-Cluster (CC): Encodes local geometry using a local Frenet-frame formulation (chord length, relative tangential deviation, curvature, and residual offsets) rather than absolute coordinates. This ensures translation invariance and compacts the sequence length by eliminating redundant separators, terminating sequences solely with an <EOS> token.

2. Geometry-Aware Feature Extraction

To bridge the modality gap between 2D images and 3D CAD sequences, the method employs a coarse-to-fine point cloud intermediate representation:

• Coarse Lifting: A single-view image is lifted into an initial coarse point cloud using a tailored version of Dens3R. To adapt this to rigid industrial parts, the authors employ Parameter-Efficient Fine-Tuning (PEFT) on the Query and Value linear layers, aligning spatial attention with CAD priors.
• Refinement: To mitigate noise while preserving sharp edges, a novel Uncertainty-Aware DGCNN (UA-DGCNN) is introduced. It predicts geometric importance scores and utilizes a Heuristic-guided Resampling strategy. This strategy balances the retention of high-curvature features (sharp edges) via a Boltzmann distribution with uniform background coverage to prevent voids in planar regions.

3. Cross-Modal Alignment and Generation

The system aligns the refined point cloud embeddings with the CAD latent space to guide generation:

• Contrastive Learning: A joint framework uses InfoNCE loss to maximize mutual information between the point cloud embeddings and the CAD sequence embeddings, ensuring the structural information from the geometry is semantically connected to the CAD codebook.
• VQ-Diffusion Generation: The aligned features serve as conditions for a VQ-Diffusion model. This model predicts discrete tokens for the CAD sequence via an absorbing state transition process.
• Compilation: A geometric kernel compiles the predicted discrete tokens into a watertight BRep model, outputting a standard STEP file.

Key Contributions

1. Hierarchical Three-Layer Codebook: A novel encoding strategy that compresses complex CAD sequences into a discrete latent space suitable for diffusion modeling, prioritizing global profiles over local details to ensure topological validity and learnability.
2. Industrial Data Curation (CAD-220K & PrintCAD): To address the scarcity of industrial data and the sim-to-real gap, the authors introduce:
   • CAD-220K: A curated subset of the ABC dataset containing over 220,000 diverse 3D CAD models with rendered images.
   • PrintCAD: A dataset of over 2,000 images of 3D-printed components captured under real-world lighting and backgrounds.
     These datasets are used to train the image-to-point-cloud module, enhancing generalization on mechanical parts.
3. Robust Alignment Framework: A conditioning mechanism that aligns 2D image-derived point clouds with CAD sequence encodings via contrastive learning, enabling the generation of topologically valid CAD sequences that compile directly into standard STEP files.

Experimental Results

Extensive experiments demonstrate that Img2CADSeq outperforms state-of-the-art methods across three settings:

• Image-Conditioned Generation: The method achieves the lowest Chamfer Distance (1.21), the lowest Ratio of Hanging Faces (2.2), and the highest Segmentation Accuracy (97.2%) compared to baselines like TripoSR, HoLa, and CADDreamer. Visually, it produces BReps with sharp edges and minimal fitting errors, avoiding the "clay-like" distortions or non-watertight artifacts seen in competing methods.
• Point Cloud-Conditioned Generation: On clean point clouds, Img2CADSeq achieves the lowest Accuracy Error (6.49) and Completeness Error (4.91), with high Precision (0.947) and Recall (0.923). It successfully recovers sharp edges and fine geometric features that fitting-based methods often smooth over or fail to trim correctly.
• Unconditional Generation: In unconditional generation tasks, the method demonstrates superior distribution similarity (lowest MMD and JSD) and diversity (highest Coverage and Novelty) while maintaining a low Invalid Rate (10.89), producing structurally plausible mechanical components.

Significance and Limitations

The paper claims that Img2CADSeq significantly advances the field of automated reverse engineering by generating editable, topologically valid BReps rather than inert meshes. By directly outputting standard STEP files, the method facilitates downstream industrial applications such as manufacturing and simulation.

The authors modestly acknowledge specific limitations:

• Single-View Ambiguity: In cases of severe occlusion, the model may hallucinate occluded regions that are topologically watertight but physically impossible to manufacture.
• Global Constraints: While local features are captured well, the method lacks explicit global constraints, leading to potential errors in strict symmetry or coaxiality due to sequence error accumulation.
• Resolution Limits: The intermediate point cloud resolution (4,096 points) may cause micro-structures like threads or tiny fillets to be smoothed out or omitted during resampling.

Future work is proposed to integrate Vision-Language Models (VLMs) for multimodal conditioning, enable interactive editing at intermediate stages, and explicitly enforce strict geometric constraints.