From Raw Agent Traces to Auditable Claims: A Reproducibility Layer for AgentDojo

In the rapidly evolving world of artificial intelligence, researchers are constantly testing how well computer programs can act on their own. These programs, known as agents, are designed to use tools, make decisions, and interact with digital environments to complete tasks. To know if these agents are truly capable or if they are easily tricked, scientists run them through rigorous benchmarks, which are like standardized tests. One such test, called AgentDojo, specifically looks at how well these agents handle tricky situations, such as when someone tries to trick them with hidden instructions. However, a major problem has emerged: while these tests produce a final score, the path to that score is often a black box. It is difficult to verify exactly which attempts were counted, how the final numbers were calculated, or whether the results reported in a research paper actually match the raw data the computer generated. Without a clear trail, it is hard to know if a high score represents genuine skill or just a mistake in how the results were tallied.

This uncertainty is the focus of a new software tool developed by independent researcher Abdullah Alharbi. His work, titled "AgentDojo Trace Audit," does not create a new test or a better agent. Instead, it builds a transparent layer of verification that sits between the raw data of the AgentDojo test and the final claims made by researchers. The goal is to turn a static report into an auditable story. The tool takes a specific, frozen version of the AgentDojo test and checks every single step of the process. It verifies that the files used are exactly the ones intended, that the rules for including or excluding data were followed perfectly, and that the final numbers in the report match the underlying records. It acts as a rigorous accountant for the experiment, ensuring that the numbers add up and that the story told by the data is consistent from start to finish.

To do this, the software first requires a user to have a legitimate copy of the official AgentDojo source code. It does not redistribute the test data itself, which keeps the process legal and focused on verification rather than data collection. Once the correct source is present, the tool begins its work by examining the digital fingerprints of every file. It calculates a unique code for each file based on its contents, ensuring that even a tiny change, like a single altered letter or a different file name, would be immediately spotted. This step confirms that the researcher is working with the exact same digital material that was used in the original study, preventing any accidental or intentional tampering with the source material.

Once the source is verified, the software moves to the heart of the audit: counting and categorizing. The original test produced thousands of records, but not all of them were meant to be part of the final analysis. The tool applies a strict set of rules to decide which records to keep and which to discard. For example, it found that out of nearly 11,800 candidate files, 140 were auxiliary cases that did not fit the main group, and one file was incomplete because it was missing key data fields. By applying these rules, the software reconstructed the exact population of 11,624 records that the original study claimed to analyze. It then checked that the breakdown of these records into different categories—such as whether an attack was successful or failed—matched the official report perfectly. In this specific test, the tool confirmed that every single one of the 11,624 records was correctly linked to its source and that the final counts for success, failure, and attack outcomes were accurate.

The real power of this tool, however, lies in its ability to find errors, not just confirm correct ones. To test its reliability, the researchers created a series of challenges where they deliberately introduced mistakes into the data, such as swapping files, deleting links between records, or changing the meaning of a result. In one set of tests involving 408 different scenarios, the tool successfully detected and classified every single error. However, it could only pinpoint the exact location of the problem in 336 of those cases. The remaining 72 failures were not because the tool missed the error, but because the way it described the location of the error was not detailed enough. The researchers then fixed this specific issue in the software's design.

After making this repair, they ran a new, more rigorous test with 360 fresh scenarios. This time, the tool was perfect. It detected, classified, and located every single one of the 360 errors without a single miss. To ensure the tool was not too sensitive and was not flagging harmless changes as errors, they also ran it against 120 benign cases, such as files that were simply renamed or compressed differently. The tool correctly ignored all of these, producing zero false alarms. This sequence of testing demonstrates that the software can reliably distinguish between a harmless change in how a file is stored and a genuine change in the scientific content.

The significance of this work is that it provides a method for turning a complex, opaque experiment into something that can be inspected line by line. Before this tool, if a researcher claimed a certain success rate, others had to take their word for it or try to manually re-run the entire experiment, which is often impossible. Now, anyone with the original source data can run this audit to see exactly how the numbers were derived. It does not tell us if the agents are safe in the real world, nor does it prove that the original test design was perfect. Instead, it guarantees that the numbers reported in a paper are consistent with the data they are based on. It ensures that the story told by the data is the same story that the data actually supports.

The researchers are careful to note the limits of their work. The tool is designed for one specific version of the AgentDojo test and cannot be automatically applied to other benchmarks or future versions without significant reworking. It also cannot detect if the original data was factually wrong or if the test itself was flawed; it only checks for consistency and internal logic. Furthermore, because the tool was developed by the same team that created the tests and the verification logic, it is not an independent external review, but rather a self-check that has been rigorously stress-tested. Despite these boundaries, the work offers a clear path forward for scientific transparency. It shows that it is possible to build a system where the claims made in research are not just statements, but verifiable facts that can be traced back to their source, one byte at a time.