🤖 AI

MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

This paper introduces MUSE, a new benchmark and evaluation framework designed to advance Text-to-CAD generation beyond simple geometric similarity by assessing complex, editable B-Rep assemblies against critical engineering criteria such as functionality, manufacturability, and assemblability.

Original authors: Xiaoyu Dong, Zhi Li, Xiao-Ming Wu

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoyu Dong, Zhi Li, Xiao-Ming Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, creative robot that can listen to your voice and draw pictures of 3D objects. If you say, "Make me a chair," it might draw a beautiful chair that looks perfect on a screen. But in the real world, a chair isn't just a picture; it's something you have to build, put together, and sit on without it collapsing.

This paper introduces a new test called MUSE to see if our current "smart robots" (AI models) can actually design real, buildable objects, not just pretty pictures.

Here is the breakdown of what they did, using some simple analogies:

1. The Problem: Drawing vs. Building

Previously, AI tests for 3D design were like a high school art class. The teacher would say, "Draw a chair," and the grade was based on how much the drawing looked like a real chair. If the legs were slightly crooked but it still looked like a chair, it got an A.

But in the real world (engineering), a chair with crooked legs is a disaster. It might wobble, break, or be impossible to build with wood or plastic. The old tests didn't check if the chair would actually work.

2. The New Test: MUSE (The "Master Builder" Exam)

The authors created MUSE, which is like a final exam for a Master Builder. Instead of just asking the AI to "draw a chair," they give it a detailed Design Specification. Think of this as a strict blueprint that says:

What it is: A wooden chair.
How to build it: Use CNC machines (like a super-precise saw).
How to put it together: The legs must slot into the seat using specific joints.
The Rules: The wood can't be too thin, or it will snap. The pieces can't overlap in a way that makes them impossible to cut.

3. The Three-Stage Filter (The Funnel)

The paper tests the AI in three strict stages, like a funnel that filters out bad designs at every step:

Stage 1: The Code Check (Can it speak the language?)
The AI has to write a computer program (a script) to build the object. If the code has a typo or a logic error, the program crashes.
- Analogy: It's like asking a chef to write a recipe. If the recipe says "add salt" but forgets to say how much, the dish is ruined before it starts.
- Result: Many AIs failed here. They couldn't even write a working script.
Stage 2: The Geometry Check (Is the shape valid?)
If the code runs, the computer builds the 3D model. Then, engineers check if the shape makes sense physically. Is it watertight (no holes)? Do the parts overlap in a weird way?
- Analogy: Imagine the chef followed the recipe, but the cake they baked has a hole in the middle or the frosting is melting into the sponge. It looks like a cake, but you can't eat it.
- Result: Even when the code worked, the shapes often had "glitches" that made them impossible to manufacture.
Stage 3: The Design Intent Check (Does it actually work?)
This is the hardest part. The AI passes the code and shape checks, but does the chair actually function? Can you sit on it? Is it stable? Can you assemble the parts?
- Analogy: The chef baked a perfect-looking cake, but it's made of stone. It looks like a cake, but it's useless for eating.
- Result: This is where the biggest drop happened. Even the smartest AIs struggled to make designs that were truly functional, manufacturable, and easy to assemble.

4. The "Rubric" Judge (The Strict Teacher)

To grade these designs, the researchers didn't just ask a human to look at them (which is slow and expensive). They built a special AI Judge (a Vision-Language Model) that acts like a strict teacher.

This judge doesn't just say "It looks good." It checks against a Rubric (a checklist).
The Checklist: "Does the leg connect to the seat? Is the wood thick enough? Are the joints tight?"
The researchers tested this AI Judge against human experts and found it agreed with them about 83% of the time. This means the computer can reliably grade the designs without needing a human to check every single one.

5. The Big Discovery

The paper found a "Failure Cascade."

If you ask an AI to design a complex, buildable object:
1. It often fails to write the code.
2. If it writes the code, the shape is often broken.
3. If the shape is perfect, the design usually fails to work in the real world (it's unstable or unbuildable).

Even the most powerful AI models (the "smartest" ones) only managed to pass the final "Design Intent" check about 20% to 50% of the time (depending on the model). The open-source models (free ones) did even worse, often scoring near zero on the final engineering criteria.

The Bottom Line

The paper concludes that while AI is great at making things look like 3D objects, it is still very bad at making things that are actually useful.

We are currently at a point where AI can draw a chair, but it cannot yet design a chair that a carpenter could build and a human could sit on safely. The MUSE benchmark is a new tool to push AI developers to stop making pretty pictures and start making real, working designs.

Technical Summary: MUSE Benchmark for Text-to-CAD Generation

Problem Statement

While Large Language Models (LLMs) have advanced text-driven 3D generation, the specific task of Text-to-CAD (generating editable, boundary representation (B-Rep) models from natural language) remains insufficient for industrial product design. Existing benchmarks primarily focus on single-part CAD models and rely on geometric similarity metrics (e.g., Chamfer Distance). These metrics fail to capture critical engineering requirements: functionality, manufacturability, and assemblability. Consequently, current evaluations cannot determine if a generated model is a viable design instance or merely a visually plausible shape. Furthermore, existing datasets often lack multi-component structures and assembly constraints, limiting their utility for real-world fabrication.

Methodology

The MUSE Benchmark

The authors introduce MUSE, a benchmark designed to evaluate the generation of complex, editable B-Rep assemblies. Unlike prior works, MUSE focuses on complete design instances defined by structured Design Specifications ( $S$ ). Each specification decomposes a high-level goal into four components:

Design Description ( $D$ ): High-level text describing the object.
Physical Assembly Graph ( $G$ ): An undirected graph where vertices represent physical components (e.g., legs, panels) and edges represent physical interfaces.
Valid Parameter Space ( $\Omega$ ): Admissible ranges for component parameters (dimensions, thickness) constrained by functional and manufacturing requirements.
Manufacturing Plan ( $M$ ): Constraints regarding material selection (e.g., Timber, PLA) and manufacturing methods (e.g., CNC Milling, 3D Printing), including tolerance limits and geometric constraints.

Dataset Construction:
The dataset of 106 design instances was curated via a "Human-in-the-loop" pipeline:

Seed Modeling: Expert designers created high-quality STEP models and corresponding CadQuery scripts.
LLM Augmentation: LLMs (Claude Opus 4.7) expanded these scripts to create diverse variants (stylistic and functional).
Engineering Views: Instead of rendered images, the system extracts precise 2D engineering views (Top, Front, Right, Isometric) with hidden lines to ensure accurate geometric assessment.
Specification Synthesis: LLMs generated structured Design Specifications based on the models, which were then reviewed and corrected by human experts.

Three-Stage Evaluation Protocol

MUSE employs a funnel-style evaluation pipeline where a model must pass each stage to proceed to the next:

Code Check: The generated CadQuery script is executed in a sandbox. Success requires the script to run without errors and export a valid STEP file.
Geometric Check: The exported STEP file undergoes four binary checks:
- Watertight: No open boundaries.
- Manifold: Valid topology.
- Self-Intersection Free: No internal overlaps.
- Overlap Free: Distinct components do not interpenetrate.
Design-Intent Alignment: Valid models are converted to engineering views and evaluated by a Rubric-based Visual Language Model (VLM) Judge. The judge uses design-specific rubrics (generated from the Design Specification, reference drawings, and engineering knowledge tables) to assess:
- Functionality: Does the design support its intended use and remain stable?
- Manufacturability: Does the geometry satisfy material and process constraints (e.g., wall thickness, tolerance)?
- Assemblability: Are the component topology and joint interfaces preserved correctly?

Key Contributions

Shift in Evaluation Paradigm: MUSE moves beyond geometric similarity to evaluate engineering usability, explicitly testing for functionality, manufacturability, and assemblability.
Structured Design Specifications: The introduction of a formal definition for Design Specifications ( $S = \langle D, G, \Omega, M \rangle$ ) provides a rigorous framework for defining complex, multi-component assemblies with valid parameter spaces.
Rubric-Based VLM Judge: The paper proposes a scalable evaluation method using a VLM judge guided by fine-grained, design-specific rubrics. This approach reduces hallucination by grounding the evaluation in engineering constraints and reference views, validated against human annotations.
Comprehensive Dataset: A dataset of 106 diverse, manufacturable, and assemblable design instances covering various materials (Timber, PLA, ABS, etc.) and processes (CNC, 3D Printing, Laser Cutting).

Experimental Results

The authors evaluated both closed-source (e.g., GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) and open-source LLMs (e.g., Qwen, Llama) on MUSE.

Failure Cascade: Results reveal a clear degradation in performance across the three stages. While some models achieve moderate success in Code Execution (e.g., GPT-5.5 at ~77%), performance drops significantly in Geometric Validity (e.g., ~69% for GPT-5.5) and further collapses in Design-Intent Alignment (e.g., ~52% for GPT-5.5).
Fine-Grained Deficiencies: Even the strongest models struggle with specific engineering criteria. For instance, the best closed-source models achieved only ~43% on Assemblability and ~48% on Manufacturability. Open-source models performed significantly worse, often failing to produce valid assemblies.
Code vs. Geometry: High code executability does not guarantee geometric validity or design intent alignment, indicating that current models lack robust 3D spatial reasoning and engineering constraint handling.
Judge Reliability: The rubric-based VLM judge showed strong correlation with human annotations (Pearson $r \approx 0.71$ at the sub-criteria level and $r \approx 0.83$ at the design-instance level), validating its use for automated evaluation.

Significance

The paper posits that Text-to-CAD has been easier to measure as shape generation than to solve as design generation. MUSE raises the bar by demanding that generated CAD models be not just "right" in shape, but "right" in engineering practice—capable of being built, assembled, and used. The benchmark exposes a core weakness in current LLMs: while they can synthesize code, they fail to coherently generate assemblies that satisfy coupled geometric and engineering constraints. By providing a realistic benchmark and evaluation framework, MUSE aims to drive future research toward models that can generate truly functional and manufacturable designs.