Technical Summary: Text2CAD-Bench

Problem Statement

While Large Language Models (LLMs) have catalyzed progress in generating parametric Computer-Aided Design (CAD) models from natural language, the systematic evaluation of these systems remains critically limited. Existing benchmarks suffer from three primary shortcomings that misalign human design intent with machine-executable representations:

1. Restricted Geometric Complexity: Current datasets predominantly consist of simple sketch-extrude primitives, lacking support for advanced modeling features essential for real-world manufacturing, such as chamfers, fillets, sweeps, lofts, and freeform surfaces.
2. Narrow Domain Coverage: Prior benchmarks focus almost exclusively on traditional mechanical parts, overlooking diverse applications in medical devices, consumer products, and architectural fabrication.
3. Insufficient Linguistic Diversity: Existing descriptions often mirror low-level command sequences rather than conveying high-level design intent. This encourages models to learn superficial "command-translation" mappings instead of achieving genuine "shape-understanding."

Methodology

The authors introduce Text2CAD-Bench, the first comprehensive benchmark designed to evaluate text-to-parametric CAD generation across a rigorously defined complexity hierarchy and diverse application domains.

Benchmark Construction

The benchmark comprises 600 human-curated examples, organized into four progressive tiers (L1–L4) and paired with dual-style natural language prompts:

• Geometric Descriptions: Mimic non-expert users by characterizing shapes via appearance, spatial relationships, and global-to-local details.
• Sequence Descriptions: Align with expert-level CAD conventions, specifying step-by-step construction logic.

The construction pipeline follows a human-in-the-loop process:

1. Design Specification: Defining target geometry and complexity levels.
2. Collaborative Authoring: Human designers specify geometry and iteratively develop CadQuery code (a Python-based API) with AI assistance, ensuring valid, watertight meshes and matching complexity levels.
3. Description Generation: Creating the dual-style prompts for each validated model.
4. Quality Assurance: Verifying executability, description-geometry correspondence, and cross-style consistency.

Complexity Levels

• L1 (Basic): Fundamental geometry with primitive shapes and basic spatial relationships.
• L2 (Intermediate): Compositional complexity involving boolean operations and standard CAD features (e.g., fillets on boolean cuts).
• L3 (Advanced): Sophisticated operations including sweeps, lofts, shells, and complex patterns, requiring mastery of path-based geometry and freeform surfaces.
• L4 (Real-World): 100 examples spanning diverse domains (industrial, consumer, medical, architectural). These evaluate generalization within application contexts (e.g., "suitable for M4 screw") while maintaining precise geometric parameters.

Evaluation Metrics

The benchmark employs a multi-metric approach to assess both code validity and geometric fidelity:

• Chamfer Distance (CD): Measures geometric similarity between generated and ground-truth meshes.
• Invalidity Rate (IR): The percentage of code samples failing to execute due to syntax errors, timeouts, or degenerate geometry.
• Intersection over Union (IoU): Measures volumetric overlap.
• VLM-based Evaluation (for L4): Since ground-truth code comparison is insufficient for contextual design quality, a Vision-Language Model (GLM4.6V) scores generated models on 8 multi-view renderings across five feature categories and overall similarity.

Key Contributions

1. Dataset Construction: The creation of a 600-example dataset addressing the lack of advanced geometric operations (chamfers, fillets, sweeps, lofts) and application diversity in prior benchmarks.
2. Dual-Style Prompt Design: The introduction of geometric and sequence prompt styles, with empirical evidence demonstrating that geometric descriptions generally yield superior generation performance on basic tasks, while sequence descriptions show advantages on complex, procedural tasks.
3. Comprehensive Evaluation: A rigorous assessment of mainstream general-purpose LLMs (e.g., GPT-5.2, Claude-4.5, DeepSeek-V3.2) and domain-specific models, revealing significant performance gaps in advanced features and real-world scenarios.

Experimental Results

Experiments conducted on Text2CAD-Bench reveal several critical findings:

• Performance Degradation with Complexity: While models perform reasonably on L1–L2 tasks, performance degrades substantially on L3. Chamfer Distance increases by 1.3–2.1×, and Invalidity Rates rise from ~15% to 70–90% as models encounter advanced features like sweeps and lofts.
• Prompt Style Sensitivity: Geometric descriptions outperform sequence descriptions on L1–L2 across multiple metrics. However, this advantage diminishes or reverses on L3, where the procedural nature of advanced operations makes sequence descriptions more effective for conveying construction logic.
• General-Purpose vs. Domain-Specific Models: A striking divergence exists between model categories. Domain-specific models (e.g., Text2CAD) often achieve low Invalidity Rates but extremely poor geometric fidelity (high CD, near-zero IoU), suggesting they generate syntactically valid but geometrically imprecise code. Conversely, general-purpose LLMs show better geometric alignment but struggle with code execution on complex tasks.
• Capability Decoupling in Real-World Scenarios (L4): Evaluation of L4 reveals that code executability, geometric precision, and design intent understanding are largely independent dimensions. No single model dominates all three; for instance, a model with the lowest Invalidity Rate may score poorly on feature-level design questions, while a model with high geometric similarity may fail to capture functional context.

Significance and Claims

The paper positions Text2CAD-Bench as a challenging target for future research, establishing that text-to-CAD remains largely unsolved beyond basic geometry. The authors claim that current state-of-the-art systems struggle significantly with advanced features and real-world application contexts, indicating a need for substantial advances in model capabilities or training data before reliable professional workflows can be supported.

Furthermore, the study highlights that output representation matters critically, demonstrating that Python-based CadQuery representations allow models to better leverage pretrained code generation capabilities compared to traditional command sequences. Finally, the benchmark underscores the necessity of multi-dimensional evaluation, as optimizing for code correctness or geometric fidelity in isolation does not guarantee success in holistic design tasks. The authors release the benchmark to drive progress in the field, acknowledging that while 600 examples cannot exhaustively represent the full CAD design space, they provide a necessary, high-quality foundation for diagnosing model limitations.