Imagine a world where you can describe a machine part in plain English, and a computer instantly draws a precise, three-dimensional blueprint for it. This is the promise of text-to-CAD systems, a rapidly advancing field where artificial intelligence translates natural language into the structured, mathematical commands that engineers use to design everything from car engines to medical devices. For these systems to be truly useful, they must do more than just sketch a shape that looks right; they must understand the exact position and orientation of every piece in space. If a designer asks the computer to move a part three inches to the right and then rotate it ninety degrees, the machine must execute those specific instructions with perfect accuracy. If it fails, the resulting blueprint could be unusable, leading to parts that do not fit or machines that cannot function. The core challenge lies in whether these intelligent systems can reliably transport a design through a requested change in its coordinate frame, a task that requires a deep, numerical understanding of geometry rather than just a visual approximation.

A researcher led by Shengyao Sun at Shanghai Jiao Tong University has developed a new way to test this specific capability, revealing a surprising weakness in how current models handle spatial transformations. They created a diagnostic tool called FrameShift-CAD, which acts like a controlled stress test for these AI systems. Instead of asking the models to generate a design from scratch, the researcher provided them with a complete, valid blueprint and then gave a single, clear instruction to move or rotate that entire design. To ensure the test was fair and rigorous, they created two versions of the instruction for every design: one that asked the model to physically move the object in a fixed space, and another that asked the model to keep the object still but describe it as if the space around it had moved. By comparing the results of these paired instructions against mathematically perfect answers, the researcher could pinpoint exactly where the models succeeded and where they failed.

The study, which examined a wide variety of simple geometric shapes and tested them across several different large language models, uncovered a stark difference in performance between moving objects and turning them. When the instruction was to translate, or shift, the design to a new location, the models were generally successful, getting the position right in about two-thirds of the cases. However, when the instruction was to rotate the design around an axis, the success rate plummeted. In a confirmatory phase of the study involving twelve different families of shapes, the models correctly executed the rotation only eight percent of the time. This gap was not a fluke; it persisted across different types of rotations, including turns around the horizontal and vertical axes, and even when the rotation was combined with a movement. The models consistently struggled to turn the object as requested, regardless of how the instruction was phrased.

Perhaps the most revealing finding was not just that the models failed, but how they failed. The researcher expected that if a model got a rotation wrong, it might simply turn the object in the opposite direction, effectively flipping the instruction. Instead, the data showed that in nearly eighty percent of the failed rotation attempts, the models did not turn the object at all. They produced a blueprint that looked exactly like the original, unrotated design. It was as if the instruction to rotate was heard but ignored, and the model simply returned the object in its starting position. This "no-operation" error pattern suggests that the models are not confused about the direction of the turn, but rather are failing to execute the turn itself. When the instructions were more complex, asking the model to both move and rotate the object, the models often managed to move the object correctly but dropped the rotation entirely, leaving the object in the new spot but facing the wrong way.

The researcher also investigated whether the difficulty stemmed from the confusing language used to describe these movements. They tested the models with instructions that used only the mathematical formulas for the rotation, without any descriptive words about "frames" or "active" and "passive" movements. They also tested instructions that used only the descriptive words without the formulas. The results showed that while the descriptive words added some difficulty, removing them did not fix the problem. Even when given the pure mathematical mapping, the models still failed to rotate the objects correctly at the same high rate. This indicates that the issue is not merely a misunderstanding of terminology, but a deeper inability to perform the geometric operation of rotation within the generated code. The models appear capable of handling linear shifts but lack the reliable mechanism to apply rotational transformations to their own output.

This discovery has significant implications for the future of automated design. It suggests that simply asking a language model to "rotate this part" is not enough to guarantee a correct result, even if the model can generate a perfect design from a text description. The study provides a clear method for diagnosing these failures, distinguishing between a model that gets the direction wrong, one that gets the magnitude wrong, and one that simply does nothing. By identifying that the primary failure mode is a failure to execute the rotation rather than a confusion of signs, the research points toward specific engineering solutions. Future systems may need to separate the generation of the shape from the application of the transformation, using a deterministic, rule-based step to handle the rotation after the shape is created, rather than relying on the language model to calculate the rotation on the fly. Until such safeguards are in place, the promise of fully autonomous text-to-CAD generation remains limited by this specific, measurable gap in spatial reasoning.