HEPTABench: A framework for portable and controlled agentic evaluations in high energy physics

In the vast, silent corridors of modern physics, scientists are no longer just looking for new particles; they are also looking for new ways to do the work of finding them. For decades, the field of high-energy physics has relied on human researchers to translate abstract theories into concrete calculations, writing complex computer code to simulate how particles might behave in giant detectors or to predict the results of collisions. Today, a new kind of worker has entered the laboratory: the artificial intelligence agent. These are not simple programs that follow a rigid list of instructions, but sophisticated systems capable of planning multi-step tasks, writing their own code, running simulations, and correcting their own mistakes when things go wrong. They promise to accelerate the pace of discovery, but they also introduce a new question: can we trust them to do the delicate, precise work of science? To answer this, researchers must move beyond simple tests and create a controlled environment where they can see exactly how these digital assistants perform, what tools they need, and where they might fail.

A team of physicists and computer scientists has built such an environment, a framework they call HEPTABench, designed to put these AI agents through their paces in the specific world of high-energy physics. Their goal was not just to see if an AI could solve a problem, but to understand how the tools it uses change the outcome. They set up a series of experiments where the same scientific task was given to different AI systems, but with one crucial variable changed: in some trials, the AI had to build its own mathematical tools from scratch, while in others, it was handed a pre-made, verified toolkit of specialized instruments. The results were clear and consistent. When the AI agents were allowed to use these specialized, domain-specific tools, their work became significantly more accurate and far more reliable. The results they produced were closer to the known truth, and the answers they gave were much more consistent from one attempt to the next.

The researchers tested this idea with two very different types of physics problems. The first was a complex calculation involving a hypothetical particle that lives for a long time before decaying. This kind of calculation is a staple of the field, requiring a chain of steps that includes predicting how the particle is created, how it travels through a detector, and how likely it is to be seen. The team asked four different advanced AI systems to perform this calculation. In the first round, the systems were given only the basic laws of physics and the experimental setup, forcing them to write every piece of code needed to simulate the particle's journey. In the second round, the same systems were given the same task but were also provided with a set of specialized tools that could handle specific parts of the calculation, such as generating particle streams or calculating decay probabilities. The difference was striking. Every single system, regardless of its underlying architecture, produced better results when it had access to the specialized tools. The numbers they calculated were closer to the correct values, and the variation between different attempts by the same system dropped dramatically. The AI agents also wrote far less code, relying on the trusted tools for the heavy lifting rather than reinventing the wheel for every trial.

The second experiment focused on a different kind of challenge: symbolic mathematics. Here, the task was to derive formulas for how particles decay, a process that requires manipulating complex algebraic expressions rather than just crunching numbers. The researchers tested a specific AI model on eight different decay scenarios. Without the specialized tools, the model struggled immensely, succeeding in only a tiny fraction of its attempts. It often produced incomplete answers or formulas that did not match the physics. However, when the same model was given access to a toolkit designed specifically for symbolic physics calculations, its performance soared. The fraction of successful attempts jumped from a mere eight percent to nearly eighty percent. This showed that providing the right tools could transform a system that was barely functional into one that was highly effective. Interestingly, a different, more powerful AI model was able to solve these problems perfectly using only its built-in capabilities, suggesting that while tools are essential for some systems, the most advanced models may eventually be able to handle these tasks on their own.

The study does more than just show that tools help; it establishes a method for proving it. By keeping every other factor constant—the task, the AI model, and the computer environment—the researchers could isolate the effect of the tools themselves. They found that the improvement was not just a matter of getting the right answer once; it was about consistency. The AI agents became more reliable, producing results that stayed close to the truth every time they ran the experiment. This reliability is critical for science, where a single lucky guess is not enough; researchers need to know that a result will hold up under scrutiny. The framework they built allows for this kind of detailed inspection, recording every step the AI took, every file it created, and every decision it made. This transparency means that if a result is questioned, scientists can go back and look at the exact path the AI took to get there, rather than just accepting a final number.

The implications of this work extend beyond the specific particles or calculations tested. It suggests a new way of thinking about how artificial intelligence will be used in science. Rather than expecting a single, all-knowing AI to do everything, the future may lie in pairing these intelligent systems with specialized, human-verified toolkits. Just as a human physicist relies on a library of established formulas and software packages to do their work, an AI agent can be made more capable and trustworthy by giving it access to the same resources. The researchers demonstrated that this approach works across different types of AI and different kinds of physics problems, from simulating particle collisions to deriving mathematical formulas. It offers a path forward where the speed and adaptability of AI are combined with the precision and reliability of human-engineered tools.

Ultimately, the work presented in this paper is about building trust. As artificial intelligence becomes more integrated into the scientific process, the community needs a way to measure its capabilities and limitations with the same rigor used for any other scientific instrument. HEPTABench provides that measuring stick. It shows that with the right setup, AI agents can be guided to produce high-quality, reproducible scientific work. The findings suggest that the most effective way to deploy these systems in the future is not to let them wander freely, but to equip them with the specific, verified tools they need to succeed. This approach ensures that the results they produce are not just clever guesses, but reliable contributions to our understanding of the universe.