neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing

1. Problem Statement

While 3D Computer-Aided Design (CAD) generation is advancing rapidly, 3D CAD editing remains a significant challenge. Editing is a critical part of the design workflow, involving refining models, repairing errors, and incorporating feedback.

• Limitations of Current Approaches: Existing benchmarks and models primarily rely on text-only conditioning. However, professional CAD engineers naturally communicate complex spatial instructions using a rich combination of modalities: speech, gestures, pointing, sketching (drawing), and direct manipulation of the 3D model within the software interface.
• The Gap: Current AI models struggle to interpret these temporally interleaved multimodal instructions and translate them into precise geometric modifications on Boundary Representation (B-Rep) models, which are the industry standard for manufacturing. There is a lack of high-quality datasets and benchmarks that capture this natural, multimodal interaction.

2. Methodology

The authors introduce neuralCAD-Edit, the first benchmark specifically designed for multimodal-instructed 3D CAD editing.

Data Collection Protocol

• Participants: Ten expert CAD engineers (8–13 years of experience) were recruited.
• Task: Designers were given existing CAD models and asked to provide editing requests of varying difficulty (Easy: ~2 mins, Medium: ~5 mins, Hard: ~10 mins).
• Multimodal Inputs: Requests were captured via a custom plugin in Autodesk Fusion, recording:
  • Video: Screen recording of the UI, mouse movements, and viewpoint changes.
  • Audio: Speech instructions.
  • Interaction: Selections, command executions (e.g., "extrude," "chamfer"), and timestamps.
  • Drawing: Two modes were tested: Draw-temp (temporary sketches that disappear) and Draw-static (persistent sketches).
• Ground Truth Generation: For each request, two edits were produced:
  1. Requestor Edit: The original designer performs the edit.
  2. Human Baseline: A different expert designer performs the edit based only on the request video/model (blind to the original edit), establishing a robust human baseline.
• Dataset Composition:
  • 192 Requests (4 modalities × 4 parameterization/construction types × 3 difficulties × 4 requests per type).
  • 384 Edits (Requestor + Baseline for each).
  • 30,000+ Events logged, including viewport renders and model states.

Evaluation Framework

The benchmark evaluates AI models against human experts using three evaluation axes:

1. Automatic Metrics: 3D Chamfer distance, Voxel IoU, and DINOv2 similarity on rendered views.
2. Human Evaluation: Five CAD experts rate edits on a 1–7 scale for Instruction Following and Model Quality. The primary metric is Acceptance Rate (fraction of edits scoring >5 on both metrics).
3. VLM Evaluation: Vision-Language Models (VLMs) are used to grade edits, comparing their correlation with human judgments.

Baselines

The paper benchmarks leading foundation models:

• Google Gemini 3 Pro (Native video support).
• OpenAI GPT-5.2 (Video frames extracted at 1 FPS).
• Anthropic Claude Sonnet 4.5 (Video frames extracted at 1 FPS).
• Note: All models use a harness to import .step files, write Python/CadQuery scripts, execute them, and iteratively refine the model based on visual feedback.

3. Key Contributions

1. First Multimodal CAD Benchmark: neuralCAD-Edit is the first dataset to capture natural, simultaneous multimodal instructions (video, speech, drawing, interaction) from expert engineers.
2. Human-Centric Data: Unlike synthetic benchmarks, this data is derived from real professional workflows, capturing the ambiguity and precision required in actual design tasks.
3. Modality Analysis: The study reveals that multimodal instructions (especially those including drawing) convey higher information density in less time than text-only requests and elicit more complex edits.
4. Comprehensive Benchmarking: It establishes a rigorous evaluation standard comparing frontier AI models against human experts, highlighting a massive performance gap.

4. Results

The results demonstrate a significant gap between current AI capabilities and human expertise.

• Performance Gap: Even the best-performing model, GPT-5.2, scored 53% lower (absolute) than the human baseline in human acceptance trials.
  • Human Baseline Acceptance: ~78% (GPT-5.2: ~25%, Gemini: ~10%, Claude: ~5%).
  • Automatic Metrics: AI models performed poorly on Chamfer distance and Voxel IoU compared to humans.
• Modality Impact:
  • Humans performed best on requests involving drawing (static and temp), as drawings provide precise spatial cues.
  • AI models struggled significantly with multimodal inputs, particularly the simultaneous interpretation of speech and drawing, suggesting a failure in multimodal alignment.
• Difficulty Scaling:
  • Humans naturally spent more time on "Hard" requests, adjusting their effort to complexity.
  • AI models (especially Gemini and Claude) did not scale their reasoning time or token usage effectively with difficulty, often failing to persist or "give up" on complex edits.
• Evaluation Reliability:
  • VLMs as Evaluators: VLMs (like GPT-5.2) correlated moderately with human ratings but tended to overestimate the quality of AI edits and underestimated human edits compared to human raters.
  • Automatic Metrics: DINOv2 similarity showed the highest correlation with human ratings among automatic metrics, though it still failed to capture geometric correctness in ambiguous cases.

5. Significance and Future Directions

• Benchmarking Standard: neuralCAD-Edit provides a necessary, agnostic benchmark for evaluating 3D editing capabilities, moving beyond text-only generation to interactive, multimodal editing.
• Research Directions: The paper identifies that current models lack compositional visual reasoning (e.g., understanding relative 3D positioning) and persistent task execution.
• Collaborative AI: The benchmark highlights the need for AI systems that can work collaboratively with humans, interpreting naturalistic, ambiguous instructions rather than requiring rigid, text-based commands.
• Cost/Efficiency: While AI is slower and more expensive (in token costs) than humans for high-quality edits, the benchmark serves as a crucial stepping stone to developing AI that can eventually match or exceed human efficiency in design workflows.

In conclusion, neuralCAD-Edit exposes the current limitations of foundation models in 3D spatial reasoning and multimodal instruction following, setting a high bar for future research in generative and editing AI for engineering design.