Technical Summary: UniCAD

Problem Statement

Computer-Aided Design (CAD) is fundamental to modern engineering, yet current research is fragmented. Existing approaches typically study tasks in isolation (e.g., reconstruction from point clouds, generation from text, or question answering) and rely on single input modalities. This siloed approach hinders cross-task knowledge transfer and fails to reflect real-world design scenarios where users provide mixed inputs (sketches, descriptions, scans). Furthermore, the field lacks a unified, large-scale benchmark that standardizes evaluation across diverse modalities and tasks, making fair comparison and the development of robust, generalizable models difficult.

Methodology

1. The UniCAD Benchmark

To address the lack of a unified evaluation standard, the authors introduce UniCAD, a comprehensive dataset and benchmark.

• Data Composition: The dataset contains 1,448,150 CAD models, integrating four input modalities: text descriptions, multi-view images, hand-drawn sketches, and point clouds.
• Representation: Unlike previous works using B-rep or command sequences, UniCAD converts all designs into executable CadQuery Python scripts. This choice ensures outputs are interpretable, editable, and programmatically verifiable.
• Data Generation Pipeline:
  • Source: Built upon DeepCAD, Text2CAD, and CAD-Recode.
  • Conversion: A deterministic, rule-based Python script converts JSON modeling operations to CadQuery code, achieving 100% execution success (unlike LLM-based conversion which suffers from hallucinations).
  • Multi-modal Alignment:
    • Point Clouds: Generated by uniform sampling from CAD mesh surfaces with normal vectors.
    • Images: Multi-view synthetic renderings from eight fixed perspectives.
    • Sketches: Generated using the Qwen-Image-Edit model to create realistic, hand-drawn style line art from renderings.
    • QA: A semi-automatic pipeline using GPT-4o generates 4-way multiple-choice questions grounded in visual geometry and program logic, verified by human annotators (>92% agreement).
• Tasks Supported: The benchmark evaluates four distinct tasks: Point-to-CAD, Text-to-CAD, Image/Sketch-to-CAD, and CAD Question Answering (QA).

2. The UniCAD-MLLM Model

The authors propose UniCAD-MLLM, a universal multi-modal large language model designed to handle these heterogeneous tasks within a single framework.

• Architecture: Based on Qwen2-VL-2B, extended with a lightweight, trainable point cloud projection module.
• Input Processing:
  • Text: Processed via native tokenizers.
  • Images: Encoded using the pretrained visual encoder.
  • Point Clouds: Processed via a dedicated encoder involving Furthest Point Sampling (FPS), Fourier positional encoding, and a linear projection layer to map 3D coordinates into the model's hidden dimension.
• Unified Prompting: The model treats modalities as optional token modules. Missing inputs are replaced by generic instructions, allowing flexible reasoning even with incomplete data.
• Output Generation:
  • Generation/Reconstruction: The model autoregressively generates executable CadQuery Python scripts.
  • QA: The model reasons over the prompt and the existing CAD program to generate textual answers.
• Training: The model is trained jointly on all tasks using cross-entropy loss, fine-tuning the language model and training the point projection module from scratch.

Key Contributions

1. UniCAD Benchmark: The first large-scale, unified benchmark for multi-modal, multi-task CAD learning, providing standardized data splits and evaluation protocols for text, image, sketch, and point cloud inputs across generation and understanding tasks.
2. UniCAD-MLLM: A single end-to-end model that unifies CAD generation and understanding in a shared geometric-semantic latent space, capable of processing mixed modalities and outputting executable code.
3. Empirical Validation: Demonstration that joint multi-modal, multi-task learning consistently outperforms specialized, single-task baselines, validating the scalability of unified CAD modeling.

Experimental Results

The model was evaluated on the UniCAD test set and the external Fusion360 benchmark using Chamfer Distance (CD) and Volumetric Intersection over Union (IoU).

• UniCAD Performance: UniCAD-MLLM achieved state-of-the-art performance across all modalities.
  • Point-to-CAD: CD 0.17, IoU 89.7%.
  • Text-to-CAD: CD 0.20, IoU 86.3%.
  • Image/Sketch-to-CAD: CD 0.17–0.22, IoU up to 89.8%.
  • CAD QA: Achieved 90.0% accuracy, significantly outperforming general-purpose VLMs like GPT-4o (77%) and Doubao-Seed-1.6 (85%).
• Single-View Reconstruction: Even when tested on single-view images (a harder task), the model outperformed specialized image-only baselines (CD 0.17 vs. 0.21 for Cadrille), demonstrating robust shape completion.
• Zero-Shot Generalization: On the Fusion360 dataset, UniCAD-MLLM surpassed existing methods in both point cloud (IoU 84.3% vs. 79.1%) and multi-view image reconstruction (IoU 83.9% vs. 62.5% for a hybrid LRM+CAD-Recode pipeline).
• Multimodal Fusion: Conditioning on both text and sketches yielded the best results (CD 0.18, IoU 88.6%), proving that cross-modal signals complement each other to reduce reconstruction errors.

Significance and Claims

The paper claims that UniCAD establishes a solid foundation for general-purpose CAD intelligence. By bridging multi-modal perception with executable design generation, the work demonstrates that a unified framework can learn shared geometric and semantic representations more effectively than isolated, task-specific models.

The authors emphasize that the use of executable Python scripts (CadQuery) as the output representation is critical, as it allows for direct human interpretation, programmatic validation, and integration into industrial toolchains. While the current dataset is limited primarily to sketch-and-extrude operations, the authors argue that this unified approach paves the way for future research into more complex geometric operations and interactive design scenarios. The release of the dataset, code, and pretrained models is intended to accelerate the development of robust CAD assistance systems.