Technical Summary: ProCAD – Proactive Agents for Robust Text-to-CAD Generation

1. Problem Statement

Current text-to-CAD systems, which translate natural language prompts into parametric CAD programs (specifically CadQuery), face a critical robustness issue. While Large Language Models (LLMs) have enabled initial progress, they typically operate under the assumption that user prompts are precise and consistent. In practice, natural language geometric descriptions are frequently under-specified (missing critical dimensions) or internally inconsistent (containing conflicting constraints).

Existing approaches, whether one-shot generation via fine-tuned LLMs or feedback-based refinement, tend to treat the user prompt as a reliable specification. When faced with ambiguity, these models often "hallucinate" dimensions to satisfy the request, leading to invalid code or geometrically incorrect models. This limits the practical utility of text-to-CAD for real-world engineering tasks where precision is paramount.

2. Methodology

The authors propose ProCAD, a proactive agentic framework designed to resolve specification issues before code synthesis. The system decomposes the task into a two-agent pipeline:

A. The Two-Agent Architecture

1. Proactive Clarifying Agent ($\pi_\phi$):

   • Role: Audits the user prompt to detect missing or conflicting constraints.
   • Mechanism: Modeled as a finite-horizon Markov Decision Process (MDP) where the environment is the user. The agent decides at each step whether to ACCEPT the current specification or ASK a targeted clarification question.
   • Objective: Maximize a reward function balancing geometric fidelity (minimizing Chamfer Distance) and communication overhead (minimizing interaction rounds and token length).
   • Strategy: The agent is trained to ask minimal, essential questions to produce a self-consistent specification, avoiding unnecessary interruptions.

2. CAD Coding Agent ($\pi_\theta$):

   • Role: Translates the finalized, clarified specification into an executable CadQuery program.
   • Mechanism: A standard text-to-code generation model fine-tuned on high-quality, unambiguous data.

B. Data Creation and Training Pipeline

To train these agents, the authors constructed a new, high-quality dataset and training strategy:

• High-Quality Text-to-CadQuery Dataset (10K samples):

  • Source: Derived from the DeepCAD dataset (Wu et al., 2021) via CadQuery programs.
  • Process: Instead of generating code from text, the pipeline renders shapes from multiple viewpoints and uses a frontier Vision-Language Model (GPT-5-mini) to generate natural language descriptions conditioned on both the images and the source CadQuery code.
  • Verification: A "generate-then-verify" loop ensures quality. Generated descriptions are checked for:
    1. Code Leakage: Ensuring no raw code snippets are copied into the text.
    2. Completeness: The description is fed back to the VLM to synthesize code; if the resulting geometry matches the ground truth (Chamfer Distance $< 2 \times 10^{-4}$), the sample is accepted.
  • Refinement: Samples failing checks are retried up to three times; persistent failures are routed to human experts.

• Training the Coding Agent (ProCAD-coder):

  • Fine-tuned via Supervised Fine-Tuning (SFT) on 1.6K carefully curated, unambiguous samples from the 10K dataset.
  • Base model: Qwen2.5-7B-Instruct.

• Training the Clarifying Agent (ProCAD-clarifier):

  • Trained via Agentic SFT on a synthetic dataset of 6,063 samples.
  • Synthesis: Ambiguous prompts are created by perturbing verified specifications (omitting dimensions or introducing conflicts).
  • Trajectories: The training data includes full agent trajectories where the agent asks questions and receives simulated user answers (using GPT-5-mini as a user simulator) to reach a corrected specification.
  • Goal: Learn to identify ambiguity and ask the minimal set of questions required to resolve it.

3. Key Contributions

1. Proactive Agentic Framework: A shift from static, one-shot generation to a dynamic, two-agent system that proactively detects and resolves specification ambiguities before code generation.
2. Curated High-Quality Dataset: The creation of a 10K text-to-CadQuery dataset with rigorous leakage and completeness checks, demonstrating that cleaner, more precise descriptions significantly improve downstream code synthesis.
3. Efficient Training with Minimal Data:
   • The coding agent achieves superior performance using only 1.6K fine-tuned samples, outperforming frontier closed-source models.
   • The clarifying agent is trained on a synthetic dataset of 6,063 trajectories to handle ambiguity effectively.
4. Robustness to Ambiguity: The system significantly reduces invalid code generation and geometric errors when faced with under-specified or conflicting prompts.

4. Experimental Results

Experiments were conducted on unambiguous and ambiguous test sets, comparing ProCAD against frontier models (e.g., Claude Sonnet 4.5, GPT-4o-mini) and prior open-source methods.

• Coding Performance (Unambiguous Prompts):

  • ProCAD-coder reduced the Invalidity Ratio (IR) from ~86.9% (zero-shot Qwen2.5) to 0.9%.
  • It achieved a median Chamfer Distance (CD) of 0.066 (scaled $\times 10^{-3}$), outperforming Claude Sonnet 4.5 (0.077) and prior methods like Text-CAD (0.097).

• Clarification Performance (Ambiguous Prompts):

  • In a two-agent setting with simulated user responses, ProCAD (Clarifier + Coder) achieved the lowest Mean CD (0.63) and Median CD (0.08).
  • It reduced the Invalidity Ratio to 0.9%, compared to 14.6% for a single-model Claude Sonnet 4.5 baseline.
  • Interaction Quality: ProCAD achieved the highest Efficiency (0.9654) and Resolution (0.9341) scores, indicating it asks fewer redundant questions and successfully resolves ambiguities.

• Generalization:

  • The system maintained high performance when the user simulator was switched from GPT-5-mini to Claude 4.5 Haiku, demonstrating robustness to out-of-distribution user behaviors.
  • Human-Centered Evaluation: In tests with real human annotators (100 samples), ProCAD-clarifier continued to outperform baselines in Efficiency, Resolution, and final geometric fidelity (Mean CD 1.28 vs. 9.72 for the next best baseline).

5. Significance and Claims

The paper claims that ProCAD represents a significant step forward in making text-to-CAD robust for practical engineering applications. Its primary significance lies in:

• Paradigm Shift: Moving from "reactive" generation (where models guess missing info) to "proactive" clarification (where models ask for missing info). This aligns better with the rigorous standards of engineering design.
• Data Quality Impact: The results suggest that the quality of the text description is a critical bottleneck. The authors demonstrate that even with a strong base model, poor or ambiguous descriptions lead to failure, whereas high-quality, verified descriptions enable even smaller models to outperform massive frontier models.
• Efficiency: The framework achieves state-of-the-art robustness with relatively small fine-tuning datasets (1.6K for coding, 6K for clarification), challenging the notion that massive scale is always required for reliable CAD generation.

The authors conclude that future work should focus on gathering large-scale ambiguity datasets from real-world human interactions and developing dedicated user-simulator models to further refine the proactive clarification process.