🤖 AI

Text-to-CAD Evaluation with CADTests

This paper introduces CADTestBench, the first test-based benchmark for Text-to-CAD that utilizes executable software tests to rigorously evaluate and guide the generation of CAD models, demonstrating that this approach can surpass the performance of existing methods.

Original authors: Dimitrios Mallis, Marco Wang, Ahmet Serdar Karadeniz, Elisa Ricci, Anis Kacem, Djamila Aouada

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dimitrios Mallis, Marco Wang, Ahmet Serdar Karadeniz, Elisa Ricci, Anis Kacem, Djamila Aouada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a boss giving a very specific instruction to a robot carpenter. You say, "Build me a wooden block with a hole in the middle and a slit cut all the way through it."

In the past, if the robot built a block that looked mostly like your description, but the slit stopped halfway or the hole was slightly off-center, it was hard to tell if the robot actually failed or just made a slight variation. Traditional ways of checking the work were like comparing two photos: "Does this picture look like that picture?" If the robot built a slightly different shape that still fit your description, the photo comparison might say "It's wrong" just because it didn't match the exact reference photo, even though it was a perfectly good block.

This paper introduces a new way to check the robot's work, called CADTESTS, and a new testing ground called CADTESTBENCH.

The Old Way: The "Photo Match"

Think of the old evaluation method as a teacher grading a student's drawing by comparing it to a single "perfect" drawing in a textbook.

The Problem: If you ask for "a red car," there are a million ways to draw a red car. If the student draws a red sports car and the textbook shows a red sedan, the teacher might mark it wrong just because they don't look identical, even though the student followed your instructions perfectly.
The Flaw: This method is too strict about the look and not strict enough about the rules.

The New Way: The "Code Test"

The authors realized that building a CAD model is actually like writing a computer program. Instead of comparing the final picture, they decided to check if the robot followed the rules by running a test suite.

Imagine giving the robot a checklist instead of a photo:

Does the object have exactly 16 corners?
Is the hole a perfect circle?
Does the slit go all the way through, or is there a thin wall left behind?

If the robot's creation passes every single item on the checklist, it gets an "A." If it fails one item (like leaving a thin wall), the test immediately says, "Fail! Here is exactly what went wrong."

How They Built the Test (The "Mutation" Trick)

The authors didn't just guess what tests to write. They used a clever trick called Mutation Analysis.

Imagine the robot's perfect blueprint is a cake.
The authors created "mutant" cakes: one where the hole is too small, one where the slit is missing, and one where the shape is a cube instead of a block.
They asked an AI to write tests that could spot these specific mistakes.
They kept refining the tests until they were so sharp that they could catch every mutant cake (the bad versions) while still letting the perfect cake pass.

This ensures the tests are checking the rules you gave, not just copying a specific picture.

What They Found

They tested this new system on several existing AI models that try to turn text into 3D designs.

The Results: The new "checklist" method (CADTESTS) was much better at spotting real errors than the old "photo matching" method. It also aligned much better with what human experts thought was a "good" design.
The Surprise: They found that if you let the AI run these tests while it was building the model (like a self-correcting mechanic checking the engine as it builds it), the AI got much better at following instructions. Even simple methods using this self-checking loop beat the most complex, expensive models that didn't use it.

The Bottom Line

This paper says: "Stop judging 3D designs by how closely they look like a reference photo. Start judging them by whether they pass a strict set of logical rules."

They built a new playground (CADTESTBENCH) where anyone can test their 3D-building AI using these logical rule-checks, and they showed that this method is fairer, more accurate, and helps AI models learn to build better designs.

Technical Summary: Text-to-CAD Evaluation with CADTESTS

Problem Statement

The field of Text-to-CAD generation has gained significant traction, driven by the capabilities of Vision Language Models (VLMs) to translate natural language prompts into parametric 3D designs. However, the evaluation of these generative models remains a critical bottleneck. Current evaluation paradigms predominantly rely on geometric similarity metrics, such as Chamfer Distance (CD) or Intersection over Union (IoU), computed against a single ground-truth reference model.

This approach is fundamentally flawed for Text-to-CAD tasks because natural language prompts are inherently ambiguous. A single prompt often allows for a space of valid design variations (e.g., different scales, rotations, or specific geometric details not explicitly constrained). Consequently, a generated model that is geometrically distinct from the reference but fully compliant with the prompt's constraints may be penalized by similarity metrics. Conversely, a model that is geometrically close to the reference but fails to satisfy specific structural or topological requirements of the prompt may pass. Furthermore, existing metrics often fail to account for invalid generations (e.g., code that fails to execute) or lack interpretability regarding why a generation failed.

Methodology

The authors propose a paradigm shift from geometric comparison to automated software testing, treating Text-to-CAD as a code synthesis problem where natural language descriptions are translated into executable parametric programs.

1. CADTESTS

The core innovation is CADTESTS, a set of executable software tests designed to verify whether a generated CAD model satisfies the geometric and topological requirements of the input prompt.

Implementation: Each test is a Python code snippet utilizing CadQuery, a library for programmatic CAD design. These scripts query the Boundary Representation (B-Rep) of the generated model to inspect properties such as topology counts (faces, edges, vertices), bounding box dimensions, volumes, center-of-mass, and spatial relationships.
Property-Based Testing: Instead of matching a specific reference geometry, CADTESTS verify invariant properties defined by the prompt (e.g., "the model must have exactly 16 vertices" or "the hole must extend through the entire height").
Interpretability: Unlike black-box similarity scores, failing tests return explicit log messages describing the specific constraint violation.

2. CADTESTBENCH

To facilitate evaluation, the authors introduce CADTESTBENCH, the first test-based benchmark for Text-to-CAD. It is constructed using an automated pipeline that synthesizes test suites for 200 samples (derived from the CADPrompt dataset), comprising both abstract and detailed prompts.

Synthesis Pipeline: The pipeline employs a Large Language Model (LLM) planner to generate test suites.
Mutation Analysis: To ensure the discriminative power of the tests, the pipeline generates CAD mutants—modified versions of the reference CAD program that deliberately violate specific prompt requirements (e.g., altering dimensions, removing features, or changing topology).
Iterative Refinement: The test suite is refined through an iterative loop. The planner executes tests on a "passing set" (valid reference models and their augmented variants) and a "mutation set" (mutants). The planner analyzes execution traces to repair existing tests or generate new ones until all tests pass on valid models and successfully detect (kill) the mutants. This process continues for multiple rounds (R=4 in the experiments) to achieve high mutation scores (>90%).

3. Evaluation Protocol

The benchmark evaluates methods using four complementary metrics:

Pass-Rate (PR): The percentage of samples where all CADTESTS pass.
Requirement Score (RS): The fraction of prompt requirement groups satisfied per sample.
Invalidity Rate (IR): The proportion of generations that fail to execute (runtime errors).
Category Accuracy: Performance broken down by test categories (e.g., Solid/Shell Validity, Topology, Dimensions, Spatial Arrangement).

Key Contributions

CADTESTS Framework: The introduction of executable, property-based tests that verify prompt compliance directly on generated geometry, moving beyond reference-dependent geometric similarity.
CADTESTBENCH: The first test-driven benchmark for Text-to-CAD, featuring an automated pipeline for test synthesis and refinement using mutation analysis. It includes 5,937 tests across 200 samples.
Comprehensive Evaluation: A thorough benchmarking of recent Text-to-CAD methods (both fine-tuned and training-free) on CADTESTBENCH, revealing limitations in current approaches regarding topological and spatial reasoning.
Test-Driven Generation Baselines: The demonstration that CADTESTS can serve as an effective feedback mechanism during generation. By integrating test execution and log feedback into a ReAct loop, simple baselines were shown to outperform existing state-of-the-art methods.

Results

Benchmark Performance: Current training-free methods (e.g., GPT-5.2, Claude-4.6-Sonnet) generally outperform fine-tuned models (e.g., Text2CAD) on CADTESTBENCH. Fine-tuned models, often trained on synthetic data restricted to specific paradigms (sketch-and-extrude), fail to generalize to the diverse constraints in CADTESTBENCH.
Planner Impact: The choice of the underlying LLM planner significantly impacts performance, with Claude-4.6-Sonnet consistently outperforming GPT-based variants across all metrics.
Feedback Mechanisms: Baselines incorporating ReAct loops (iterative code execution and error correction) achieve lower invalidity rates and higher pass rates.
Test-Based Baselines: The proposed CADTests and CADTests + Log baselines achieved the highest performance. Specifically, the "CADTests + Log" approach using Claude-4.6-Sonnet achieved a Pass-Rate of 0.810 on abstract prompts and 0.625 on detailed prompts, surpassing all other methods.
Human Alignment: A human study involving expert annotators demonstrated that CADTESTS-based evaluation aligns significantly closer with human judgment (Cohen's $\kappa$ = 0.938 agreement on pass-rate) compared to geometric metrics (Chamfer Distance) or VLM-based scores (AUC ~0.66 vs. 0.928 for CADTests).

Significance and Claims

The paper claims that CADTESTS offers a more rigorous, interpretable, and reference-free evaluation paradigm for Text-to-CAD. By treating CAD generation as code synthesis, the authors argue that test-based evaluation correctly handles the ambiguity of natural language prompts by validating specific constraints rather than enforcing geometric similarity to a single reference.

The authors emphasize that while automated testing is standard in software engineering, its application to CAD has been overlooked. They demonstrate that:

Mutation Analysis is effective for synthesizing high-quality test suites that capture prompt specifications while remaining robust to valid geometric variations.
Test-Driven Generation is a viable strategy, where the feedback from test execution allows models to iteratively refine their outputs, often surpassing methods that rely solely on visual feedback or static prompting.
Current Limitations: Despite improvements, current LLM planners still struggle with precise topological reasoning, dimensional constraints, and spatial arrangements, as evidenced by lower accuracy in these specific test categories.

The work concludes that CADTESTBENCH provides a necessary foundation for advancing Text-to-CAD research, offering a path toward more reliable and verifiable generative design systems. The authors acknowledge that, like any test suite, CADTESTS may not cover every possible edge case (the "test coverage" limitation), but the iterative refinement process significantly mitigates this risk.