🤖 AI

Large Language Models and their Awareness of Mechanics and Spatial Geometry

This paper introduces MecEng, a fully automated benchmark that evaluates the mechanical engineering awareness of 34 large language models by testing their ability to generate simulation-ready multibody system models from textual descriptions, revealing that while current models show rapid improvement and high success rates on rigid-body tasks, they still struggle with complex flexible multibody systems.

Original authors: Johannes Gerstmayr, Sebastian Weyrer, Tobias Möltner, Peter Manzl, Michael Pieber

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Johannes Gerstmayr, Sebastian Weyrer, Tobias Möltner, Peter Manzl, Michael Pieber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where machines are not just built by engineers with blueprints and wrenches, but also designed by computers that can read a sentence and understand how the physical world works. For decades, computers have been excellent at processing language, translating text, or writing simple computer code. However, a new frontier has emerged: can these artificial intelligence systems truly grasp the laws of physics? Specifically, can they understand mechanics—the study of how solid objects move, bend, and interact—and the complex geometry required to build them? This is the question researchers at the University of Innsbruck and the University of Augsburg set out to answer. They wanted to know if modern AI could take a written description of a machine, like a crane or a robot arm, and turn it into a working digital simulation that obeys the same rules of motion and force as a real-world object.

To find out, the team created a rigorous test called MecEng, a fully automated benchmark designed to challenge artificial intelligence on mechanical engineering tasks. The researchers fed large language models, which are the powerful AI systems behind many modern chatbots, a series of text descriptions ranging from simple setups to complex machinery. These descriptions included details about mass, size, joints, and how different parts should connect. The AI's job was to write the specific computer code needed to build a digital model of that machine. The catch was that the model had to be perfect. It wasn't enough for the AI to produce a rough sketch or a visually appealing image; it had to generate a precise, mathematically accurate simulation that could run without crashing and produce the correct physical behavior. The researchers then compared the AI's work against expert-built "ground truth" models, checking everything from the shape of the parts to the way they moved over time.

The results revealed a landscape of rapid progress mixed with persistent limitations. When the tasks involved simple, rigid machines—objects that do not bend or stretch, like a swinging pendulum or a set of connected rods—the best AI models performed remarkably well. The strongest proprietary model, Claude-Opus-4.8, achieved a success rate of 91.4%, while the best open-weight model, qwen3.6:27b, reached 82.1%. This suggests that these systems have developed a strong intuition for basic mechanical concepts. They can correctly identify which parts are needed, how to connect them, and how to assign physical properties like weight and stiffness.

However, the difficulty increased sharply when the researchers introduced flexibility. In the real world, many machine parts bend, twist, and deform under pressure, such as a wind turbine blade or a flexible robotic arm. To simulate these, the AI had to not only describe the shape but also generate a detailed 3D mesh—a digital net of tiny triangles that defines the object's volume—and then calculate how that shape would vibrate and move. Here, the performance dropped significantly. While the top proprietary models managed to create correct parts about 82 percent of the time, the success rate for assembling these flexible parts into a complete, working machine simulation fell to around 65 percent for the best models. The open-weight models struggled even more, with the average success rate for the group hovering near 25 percent for the most complex assemblies, though individual models like gemma4:26b performed significantly worse at just 5%.

The study pinpointed exactly where the AI tends to fail. It is not usually a lack of understanding of the big picture; the models generally know what a machine is supposed to look like. Instead, the errors are often in the fine details. The AI might get the general shape of a part right but miss a specific chamfer, or it might assign the wrong material density, leading to incorrect weight calculations. In some cases, the AI would generate a model that looked correct but contained a subtle geometric error that caused the simulation to crash or produce impossible physics. The researchers found that these errors were not random; they often stemmed from the AI failing to follow specific instructions about how to order the parts or how to define the connections between them.

Interestingly, the researchers discovered that the way the AI was prompted mattered less than the model's inherent capabilities. Changing the wording of the instructions or asking the AI to "think" through the problem step-by-step did help in some cases, but it also dramatically increased the time and computer power required to get an answer. For the most complex tasks, the "thinking" process sometimes made the AI slower without guaranteeing a better result. The study also showed a clear trend over time: newer models, released in 2025 and 2026, consistently outperformed older ones, suggesting that the ability to understand mechanical engineering is improving rapidly as these systems are trained on more data.

Despite the impressive strides, the researchers concluded that current artificial intelligence is not yet ready to replace human engineers for complex design work. While the systems can handle simple machines with high reliability, they remain prone to subtle, hard-to-detect errors when faced with the intricate geometry and physics of flexible systems. The study highlights that while AI has learned to mimic the language of engineering, it has not yet fully internalized the physical laws that govern the real world. The path forward involves refining these models to reduce those small but critical mistakes, moving them from being capable assistants to becoming reliable partners in the design of the machines that shape our future.

Technical Summary: Large Language Models and their Awareness of Mechanics and Spatial Geometry

1. Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in code generation and mathematical reasoning, their capability to understand mechanical engineering awareness—specifically the comprehension of spatial relationships, exact geometries, physical parameters, and constraints within MultiBody Systems (MBS)—remains unquantified. Existing generative approaches for 3D geometry (e.g., NeRFs, Gaussian Splatting) prioritize visual aesthetics over the deterministic, watertight, and parametric constraints required for Finite Element Method (FEM) simulation. Furthermore, current benchmarks often lack the rigor to distinguish between logical syntax errors and fundamental mechanical reasoning failures in spatial assembly. There is a need for a benchmark that evaluates LLMs on the creation of simulation-ready mechanical models, ranging from rigid bodies to flexible multibody systems requiring exact 3D geometry generation and model order reduction.

2. Methodology: The MecEng Benchmark

The authors introduce MecEng, a fully automated benchmark designed to evaluate LLMs on the creation of multibody simulation models from parameterized textual descriptions. The framework utilizes the Exudyn multibody simulation code and Netgen for geometry generation.

2.1 Task Structure and Difficulty Levels

The benchmark comprises 84 generic tasks across three difficulty levels:

Level 1: Simple mechanical models involving mass points and single rigid bodies connected by springs and distance constraints (e.g., single mass oscillators, slider-crank mechanisms).
Level 2: Medium-complex rigid-body systems with classical joints (prismatic, revolute, rolling) and contact mechanics (e.g., Newton's cradle, double pendulums).
Level 3: Flexible multibody systems, split into two phases:
- 3a (Part Generation): Generation of 3D geometries converted into tetrahedral finite element meshes with joint interfaces and Hurty–Craig–Bampton model order reduction.
- 3b (Assembly): Assembly of the generated flexible parts into a complete MBS, testing the correct connection of interfaces.

2.2 The Generation Pipeline

The workflow transforms a textual description into a functional Python simulation model:

Context Generation: The LLM selects necessary simulation components from a predefined list (Retrieval-Augmented Generation approach) to ensure the correct topology.
Code Generation: The LLM generates Python code for the simulation model (Exudyn) or part generation (Netgen/OCCT).
Ground Truth Generation: A parallel pipeline generates the expert "ground truth" model using the same randomized parameters.
Verification: The LLM output is compared against the ground truth via a multi-stage evaluation pipeline.

2.3 Evaluation Metrics

The framework employs a hierarchical verification process:

Structural Verification:
- Component Selection: Checks if the correct simulation parts were chosen.
- Graph Isomorphism: Verifies system topology (joints, connections) and node annotations (mass, inertia, stiffness) using networkx.
Numerical Verification:
- Solver Success: Confirms the simulation runs to completion without kinematic or integration errors.
- Solution Match: Compares time-series output (body coordinates) against ground truth using a Frobenius norm with a tolerance of $10^{-5}$ .
Part-Specific Metrics (Level 3):
- Mesh Match: Validates per-vertex distance between generated and ground truth meshes.
- Eigenfrequencies Match: Compares the first three eigenfrequencies to verify stiffness and mass matrices.
- Mass and Material Match: Verifies physical properties.

3. Key Contributions

Quantification of Mechanical Engineering Awareness: The paper provides the first systematic quantification of LLM capabilities in spatial geometry and multibody dynamics, distinguishing between code executability and conceptual mechanical correctness.
Fully Automated Simulation Pipeline: A novel pipeline that generates exact 3D solid geometries and finite element meshes from text, bypassing voxel hallucinations and ensuring watertight, simulation-ready outputs.
Comprehensive Benchmarking: Evaluation of 32 open-weight and 2 proprietary LLMs (Claude-Opus-4.8, GPT-5.2) across 1,000+ test cases, including rigorous analysis of parameter dependencies (temperature, reasoning, model size, release date).
Differentiated Error Analysis: The framework separates coding errors (syntax, component selection) from mechanical reasoning errors (topology, physical parameters), offering granular insights into failure modes.

4. Results

4.1 Rigid-Body Performance (Levels 1 & 2)

Proprietary Models: The strongest proprietary model (Claude-Opus-4.8) achieved an 91.4% overall success rate, with GPT-5.2 at 86.0%.
Open-Weight Models: The best open-weight model (qwen3.6:27b) achieved an overall success rate of 82.1%, with gemma4:31b and qwen3.5:27b very close behind. The top 8 open-weight models averaged 73.0% overall success.
Performance Drop: A significant drop was observed between "topology match" (~~88.6%) and "numerical solution match" (~~77.5%), indicating that while LLMs capture the general mechanical structure, they frequently fail in fine-grained parameter selection (e.g., inertia, loads).
Temporal Trends: Performance correlates strongly with release date, with models from mid-2024 and later showing marked improvements. Open-weight models generally trail proprietary models by a 6–12 month "delay" in performance parity.

4.2 Flexible MBS Performance (Level 3)

Increased Difficulty: Flexible tasks remain considerably harder. The best proprietary model (Claude-Opus-4.8) achieved a 65.4% overall success rate for flexible MBS assembly.
Part Generation Bottleneck: The primary failure point is the generation of correct flexible parts (meshes, interfaces). For open-weight models, the average success rate for part generation was 50.7%, dropping to 24.6% for the final assembly.
Specific Failures: Models struggled with specific geometric features like chamfers on bearing blocks and correct reference position offsets for assembly (e.g., "flexible table" task).

4.3 Influence of Hyperparameters

Temperature: Low temperatures (0.0) generally yield better results for non-reasoning models. High temperatures degrade performance for mechanical tasks.
Reasoning (Thinking): Activating "thinking" modes (e.g., in GPT-OSS, Gemma 4) significantly reduces errors in kinematics and inertia (by factors of 3–5) but drastically increases token count and inference time (up to 10x). For some models (Qwen), reasoning degraded performance.
Prompt Design: The pipeline showed moderate insensitivity to prompt variations, suggesting robustness in the underlying model capabilities rather than prompt engineering.

5. Significance and Claims

The paper claims that MecEng represents a significant step forward in evaluating LLMs, moving beyond simple code generation to engineering-grade model synthesis.

Current State: The results indicate that current LLMs possess rapidly improving but still error-prone mechanical engineering awareness. While they can often construct the "skeleton" of a mechanical system, they frequently fail in the precise "flesh" (parameters, geometry, and constraints) required for valid simulation.
Limitations: The authors note that the sequential nature of Level 3 tasks (where a single part error invalidates the entire assembly) mirrors real-world engineering constraints but exacerbates the difficulty for LLMs.
Future Outlook: The paper suggests that future agentic frameworks with self-validation capabilities (iterative model improvement based on simulation outputs) may be necessary to overcome current error rates. The benchmark serves as a foundational tool to track the evolution of "mechanical awareness" in AI, highlighting that while progress is evident, reliable autonomous engineering design remains a challenge.

The study concludes that while LLMs are becoming powerful tools for mechanical engineering, they are not yet fully reliable for autonomous, error-free simulation model creation without human oversight or iterative correction mechanisms.