Reproducibility and Transparency Audit of Parallel Runtime Performance Reporting in Open-Source Benchmarks: A Case Study of the Rust, Julia, Haskell, and Python Ecosystems

In the world of computer science, researchers constantly compare different programming languages to see which one handles heavy lifting better. Imagine a team of workers trying to move a massive pile of boxes. Some teams use a single strong person, while others use many people working together. In the digital realm, this "working together" is called parallel performance, where a computer splits a difficult task across multiple processors to finish it faster. Scientists use these comparisons to decide which language to use for everything from weather forecasting to web services. However, for these comparisons to be trustworthy, the researchers must provide a complete recipe of exactly how they ran their tests. They need to list the specific computer they used, the exact version of the software, how many workers they assigned to the job, and how they measured the time. Without these details, a claim that one language is faster than another is just a story, not a scientific fact.

A researcher at University College London, Magnus Bright, decided to investigate whether scientists are actually providing these necessary details. Instead of running new tests on computers, Bright conducted a systematic audit of 160 published studies involving four popular programming languages: Rust, Julia, Haskell, and Python. The goal was not to see which language won, but to check if the winners were declared fairly. The study looked at papers published between 2015 and 2025, examining whether the author included the specific information needed for someone else to repeat the experiment and get the same result. The findings reveal a landscape where some details are shared freely, while others are often missing, leaving the scientific record incomplete.

The audit found that researchers are generally good at reporting the basics. Most papers clearly stated the size of the data they were processing and how many workers or threads they used to process it. These are the most obvious parts of the experiment, and omitting them would make the results impossible to understand. However, the study uncovered significant gaps in the more subtle, yet critical, details. For instance, only about one-third of the studies reported how they handled the initial "warm-up" period. This is a crucial step for certain languages that need to prepare themselves before running at full speed; without knowing how long this preparation took, it is impossible to know if the reported speed was real or just an artifact of the setup. Similarly, fewer than one in four studies explained how they placed their workers on the computer's processors. This placement can drastically change how fast a task runs, yet most authors left this information out.

The situation was even more uneven when it came to statistical rigor. In many fields of science, researchers report not just an average result, but also how much that result varies. This variation tells the reader if the difference between two languages is real or just random noise. In this audit, only about 40 percent of the studies included any measure of this variation. Without it, a reader cannot tell if a reported speed difference is meaningful or if it could have happened by chance. The study also looked at whether the actual code and data were available for others to check. While nearly 70 percent of the studies provided the source code, very few provided the complete environment needed to run it, such as a pre-packaged digital container that ensures the software behaves exactly the same way on a different machine. Only about 9 percent of the studies offered such a container, meaning that even when code is available, it is often difficult to use it to verify the results.

The researchers also noticed that the quality of reporting depended heavily on where the study was published and which language community produced it. Studies appearing in venues dedicated to high-performance computing systems were the most thorough, often including detailed hardware specifications and configuration settings. In contrast, studies in general computing venues were much less likely to include these details. Furthermore, the language communities themselves showed different habits. The communities around Rust and Julia tended to be more careful about reporting software versions and compiler settings, likely because their tools are designed to make such details easy to track. The Haskell community showed the lowest overall reporting scores, while Python studies often failed to report how many times they ran their tests. The study suggests that these differences are not just about individual diligence, but about the norms and expectations of the specific groups and journals involved.

When the researchers looked at why authors left out these details, they found that the reasons were often practical rather than malicious. Many authors cited strict page limits in conference papers as the reason for omitting long lists of technical specifications. Others assumed that certain details were so obvious to their peers that they did not need to be written down. Some authors simply prioritized the final results over the method, believing that the story of the discovery was more important than the technical recipe. The study also highlighted a legal and ethical blind spot: in more than half of the cases, it was unclear whether the data or code used in the benchmarks could be legally reused by others. This lack of clarity makes it difficult for future researchers to build upon past work without risking legal trouble.

To address these issues, the study proposes a simple, minimum checklist for anyone publishing a comparison of programming languages. This checklist asks authors to list the specific computer hardware, the exact software versions, how the parallel workers were configured, the size and source of the data, how many times the test was run, and the statistical measures used to analyze the results. It also asks for the code and data to be shared in a way that allows others to run them easily. The study concludes that the credibility of a performance claim depends entirely on the documentation that supports it. A speedup number without context is just a number; with the right details, it becomes a reliable piece of scientific knowledge. By adopting these clearer standards, the community can ensure that the substantial effort put into performance research yields results that can be trusted, understood, and built upon by everyone.