Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

In the world of engineering, building a bridge, a power grid, or a heating system requires more than just writing code that the computer can read. It demands that the digital model behaves exactly like the real physical thing it represents. For decades, engineers have used a specialized language called Modelica to describe these complex systems. Unlike standard programming, where a computer follows a strict list of instructions from top to bottom, Modelica works by stating relationships between quantities, like how pressure relates to flow or how voltage relates to current. The computer then figures out the order of operations to solve these relationships. This flexibility is powerful, but it creates a unique problem for artificial intelligence. An AI can easily write Modelica code that looks correct and even runs without crashing, yet still describe a physical system that violates the laws of physics or fails to meet the specific needs of the project. The machine has followed the rules of syntax, but it has missed the point of the engineering.

This gap between a model that simply works and a model that is truly correct is the central challenge addressed by a new study from researchers at TU Dresden. The team, led by Wang Zizhe, set out to see if AI agents—programs that can think, act, and use tools—could be trusted to fix, build, and tune these physical system models on their own. They found that while current AI tools are capable of generating code, they often lose track of the original engineering goals as they make changes, or they rely on test results from old versions of the model that no longer apply. To solve this, the researchers built a new system called Pufibara. This system acts as a strict supervisor that keeps a permanent record of what the project requires, ensuring that every time the AI changes the model, it re-checks the new version against the original goals. They tested this system against a leading commercial AI coding tool using a massive collection of 232 different engineering tasks. The results showed that the new system not only solved more problems but did so using significantly fewer computing resources, proving that how an AI is guided is just as important as the intelligence it uses.

The researchers approached this problem by recognizing that physical engineering is different from writing a standard computer program. In a typical software project, if a program runs without errors, it is often considered a success. In physical modeling, a model can run perfectly and still be wrong. For example, an AI might generate a model of a water pump that compiles and simulates smoothly, but if the model predicts that water flows uphill without a pump, it has failed the engineering test. The core difficulty lies in the iterative nature of the work. As an AI agent tries to fix a model, it might change a parameter, run a simulation, see a result, and then change another parameter. Without a careful memory system, the agent might forget that the first change was necessary, or it might mistakenly believe that a simulation result from the old version of the model still applies to the new one. This confusion can lead the agent to submit a final model that looks good on the surface but fails to meet the specific physical requirements of the task.

To prevent this, the team designed Pufibara with a specific architecture that treats engineering requirements as persistent obligations. Imagine a project manager who keeps a checklist of every rule the building must follow, from the strength of the steel to the flow of the air. Every time the architect makes a change, the manager checks the new design against that same checklist, ignoring any old test results that no longer match the current design. Pufibara does exactly this for AI agents. It maintains a "ledger" of engineering requirements that stays active throughout the entire process. It links every simulation result directly to the specific version of the model that produced it. If the model changes, the system knows that the old results are no longer valid and forces the agent to re-evaluate the new version. Crucially, the system requires the agent to make an explicit decision to submit a final answer. The AI cannot simply stop because it ran out of time or because the code compiled; it must actively declare that it has gathered enough evidence to prove the model meets all the engineering rules.

To test whether this approach actually works, the researchers needed a fair and realistic way to measure performance. They could not simply use existing public models, because an AI might have already seen them during its training and simply memorized the answers. Instead, they created a new benchmark called the Modelica Agent Workflow Benchmark. They started with real, working engineering models and then introduced specific faults, new design requirements, or tuning targets to create 232 unique challenges. These tasks ranged from fixing broken models to building new ones from scratch and adjusting parameters to meet specific performance goals. The benchmark included a hidden "evaluator" that acted as an independent judge. This judge did not see the AI's internal thought process or its intermediate steps; it only looked at the final model submitted by the agent and checked if it passed a strict set of physical and behavioral tests. This ensured that the AI was being tested on its ability to solve the problem, not on its ability to guess what the test looked like.

The study compared the new Pufibara system against Claude Code, a well-known commercial AI coding assistant, using two different large language models as the "brain" behind the agent. The results were clear and consistent. Across all 232 tasks, Pufibara successfully completed more problems than the commercial tool. When using one specific AI brain, Pufibara solved 202 tasks while the other tool solved 185. With a different AI brain, Pufibara again solved 202 tasks compared to 187 for the competitor. The difference was most pronounced in tasks that required building a model from scratch, where Pufibara solved significantly more problems than the other system. Beyond just solving more problems, Pufibara was also much more efficient. It used roughly 76% to 82% fewer computing tokens—the basic units of information the AI processes—and completed the tasks in significantly less time, with some runs taking up to 58% less time than the competitor.

Perhaps the most important finding was not just that Pufibara solved more problems, but that it solved the right kind of problems. The researchers discovered that the competing tool often produced models that were executable and passed basic checks but failed the deeper physical requirements. In a set of difficult tasks, the commercial tool submitted models that ran without errors but did not behave correctly according to the engineering rules in 21 out of 38 cases. Pufibara made this mistake only four times. This suggests that the commercial tool was often satisfied with a model that simply worked, whereas Pufibara's strict requirement to tie evidence to the current version of the model and to explicitly verify all engineering obligations prevented it from submitting flawed solutions. The study shows that the structure of the AI's workflow—how it remembers, checks, and decides—is just as critical as the intelligence of the language model itself.

The implications of this work extend beyond just one type of engineering language. The research demonstrates that for AI to be truly useful in complex, real-world fields like physics and engineering, it cannot just be a code generator. It must be an agent that understands the difference between a program that runs and a system that is physically correct. By keeping a persistent record of requirements and forcing the AI to re-verify its work against those requirements every time it makes a change, the system ensures that the final output is trustworthy. The researchers note that while their results are strong, they are specific to the tasks and conditions they tested. They do not claim to have solved the problem of AI in engineering forever, but they have provided a clear blueprint for how to build AI systems that are less likely to make dangerous mistakes. As the field moves forward, the focus will likely shift to testing these methods on even more complex industrial problems and exploring how they work with different types of AI models, ensuring that the tools of the future are as reliable as the systems they are designed to build.