LLM-based Visual Code Completion for Aerospace Geometric Design

Technical Summary: LLM-based Visual Code Completion for Aerospace Geometric Design

Problem Statement
The aerospace industry prioritizes safety and explainability, leading to a lag in the commercial adoption of Large Language Models (LLMs) for geometric design, despite recent advancements in Visual Language Models (VLMs) and visual code completion. Currently, no publicly announced LLM-based geometric design copilot systems are in commercial use by aerospace Original Equipment Manufacturers (OEMs). There is a lack of scientific evidence regarding the effectiveness and safety of LLMs in this high-stakes domain. Furthermore, existing visual programming environments (e.g., Grasshopper for Rhinoceros 3D) require engineers to construct complex node graphs using low-level geometric primitives, which is time-consuming and prone to error when designing complex aerospace structures like wings.

Methodology
The authors propose a LLM-based visual programming copilot application designed to assist aerospace engineers in creating wing geometry designs within the Grasshopper environment. The system integrates three core components:

1. Wingbuilder Library: A new Grasshopper plugin library containing 54 custom components grouped into seven domain-specific categories (e.g., airfoil construction, wing assembly, part factories). Unlike standard Grasshopper primitives, Wingbuilder exposes high-level domain objects (e.g., "spar," "rib," "dihedral") allowing engineers to work in their native vocabulary.
2. Aerospace Visual Programming Dataset (AVPD): A dataset comprising 18 expert-designed tasks of varying difficulty (easy, medium, hard) using the Wingbuilder components. Each task includes a starting canvas with hidden components, a free-text task description, and a ground-truth "gold answer" canvas.
3. ReAct-based Copilot Architecture: The system utilizes GPT 5.4 (a commercial LLM) employing a modified ReAct (Reasoning and Acting) methodology.
   • Input: The LLM receives a screenshot of the current Grasshopper canvas, a system prompt describing available Wingbuilder components, and a natural language description of the engineering task.
   • Process: Unlike other agentic approaches that render intermediate 3D models for visual analysis, this method skips the visual analysis step. Instead, it relies on the LLM's pre-trained knowledge of Grasshopper visual component layouts. The LLM iteratively generates "thoughts" regarding necessary actions and selects specific actions (e.g., adding a component, connecting wires) via an API.
   • Output: The system executes Grasshopper scripts to modify the canvas, providing observations back to the LLM until the task is deemed complete. The user can preview, accept, or reject suggestions at any time.

Key Contributions

• Wingbuilder Library: The release of an open-source geometry abstraction library specifically designed to support visual programming for aerospace wing design and LLM automation.
• Aerospace Visual Programming Dataset (AVPD): The creation of a benchmark dataset with 18 tasks and ground truth solutions tailored for evaluating LLM-based geometric visual code completion in a specialized domain.
• Evaluation Framework and Insights: A user trial involving two experienced aerospace engineers from a major aircraft manufacturer, providing both quantitative (task completion time, success rate) and qualitative (user satisfaction, trust) metrics. The paper also offers best practices and lessons learned for developing aerospace-domain LLM applications.

Results
The user trial yielded the following findings:

• Task Difficulty Correlation: The copilot was most beneficial for "hard" difficulty tasks (approx. 8 minutes). In these scenarios, participants using the copilot were faster (173s vs. 259s without copilot) and had a higher completion rate (5/6 vs. 2/6).
• Performance on Easy Tasks: For "easy" tasks (approx. 2 minutes), the copilot slowed participants down significantly (78s vs. 34s) due to the latency of the LLM inference.
• Inference Latency: The LLM ReAct process averaged 113 seconds per solution, with an average step time of 9 seconds and 12 steps per solution. This latency limits the tool's utility for quick, simple edits.
• User Feedback: Participants found the suggestions helpful, particularly for overcoming initial hurdles in complex tasks. However, they noted that the ReAct reasoning traces were sometimes confusing and the suggestions were not always easy to understand. Despite these issues, participants expressed a willingness to use the tool in the future.

Significance and Claims
The paper claims to demonstrate that LLM-based visual programming copilots can be effective for complex aerospace geometric design tasks, provided the latency is acceptable for the task's complexity. It highlights that:

• Domain Abstraction is Critical: Using high-level domain-specific components (Wingbuilder) rather than low-level geometric primitives is essential for enabling LLMs to reason effectively about aerospace design.
• ReAct without Visual Rendering: A ReAct methodology that relies on pre-trained LLM knowledge of visual layouts, rather than generating and analyzing intermediate 3D renders, is a viable and efficient approach for visual code completion in CAD.
• Safety and Explainability: While the tool showed promise, the authors acknowledge that the current state of open-source LLMs is insufficient for this specific ReAct workflow, necessitating the use of powerful commercial models (GPT 5.4) for viable accuracy. They posit that future work should explore ensembles of fine-tuned open-source LLMs to achieve similar performance.

The authors conclude that while the current inference speeds limit the tool to complex, time-consuming tasks, the paradigm shift toward LLM digital assistants in the aerospace industry is plausible, provided future improvements in model efficiency and hardware.