Can it be checked? A cross-sectional meta-research audit of access to, and reporting in, machine learning studies of pharmaceutical manufacturing and quality control

In the factories that make modern medicines, machines now do more than just mix and bottle. They watch, they measure, and they predict. Using a type of computer intelligence called machine learning, these systems can spot a defective tablet from a photograph, predict how a drug will dissolve, or monitor the delicate conditions inside a giant tank where living cells grow. This technology has become a standard tool for ensuring that every pill and injection is safe and effective. But for these smart systems to be trusted by the people who regulate them and the doctors who prescribe them, the scientists who build them must be able to prove their work. Just as a chef cannot claim a dish is delicious without showing the recipe and the ingredients, a scientist cannot claim a computer model works unless they show the data it learned from, the code that runs it, and proof that it works on new, unseen examples.

A new study asks a simple but critical question: when scientists publish these machine learning studies about making medicines, can anyone outside the lab actually check their work? The researcher, an independent scholar named Phani Kumar Balagam, set out to audit the published record of this field from 2019 to 2025. They wanted to know two things. First, could a regular person, without a special library subscription, actually read the full paper? Second, if they could read it, did the paper contain the specific details needed to repeat the experiment and verify the results? The answer, the study found, is that for most of these papers, the answer to both questions is no.

To find out, the researcher searched five major scientific databases for thousands of papers that described using machine learning to make or test human drugs. After filtering out reviews, preprints, and studies that did not fit the criteria, they were left with 553 papers that were eligible for the study. The first hurdle was access. The study checked whether each paper was available to the public for free. Only 186 of the 553 papers, or about one-third, could be read without a subscription. The rest were locked behind paywalls or could not be retrieved at all, even though some were marked as open access, often because of technical barriers like automated security checks that blocked the download. This means that for two out of every three studies in this field, a curious reader cannot even see the full text.

The researcher then focused on the 186 papers that were readable. They looked closely at whether these papers reported the four essential ingredients needed to verify a result: the raw data, the computer code, a clear description of how the data was split into training and testing groups, and a final performance score based on data the model had never seen before. This last part is crucial; without it, a model might just be memorizing the answers rather than learning the rules. The study found that while almost all the papers mentioned how they split their data, very few shared the actual data or the code. Only about one in four papers made the data public, and only about one in ten made the code public.

The situation was even worse when looking for the complete package. The study defined a "checkable" paper as one that shared the data, the code, the split, and the final test result all at once. Out of the 186 readable papers, only two met this standard. That is just over one percent. In other words, almost none of the published studies in this field provide enough information for an independent person to re-run the analysis and confirm the headline number. The researcher noted that this lack of transparency was not due to the nature of the data itself. Even when the data came from small laboratory experiments rather than massive industrial factories, it was rarely shared. Similarly, the lack of shared code was not because the code was secret; it was simply not provided.

The study also examined whether these problems were getting better over time or if they varied depending on the type of journal the paper was published in. The results showed no significant improvement in the rate of sharing data or code over the years, and the reporting quality was the same across different types of scientific journals. The researcher used a large language model, a type of artificial intelligence, to read and score the papers, a method they verified by having the model check the same papers twice. The model was able to find the specific sentences in the papers that supported its judgments, proving that the lack of information was a real gap in the reporting, not a misunderstanding by the reader.

The conclusion is stark. While machine learning is becoming routine in pharmaceutical manufacturing, the published record of this work is largely opaque. A third of the papers cannot be read by the public, and of the ones that can be read, almost none contain the full set of details required to verify the results. The researcher suggests that the solution does not require authors to do anything new or difficult. They simply need to share the data and code they already have and clearly state which data produced their main result. By asking for what authors already possess, the scientific community could close the gap between what is published and what can be checked, ensuring that the smart machines making our medicines are as reliable as they claim to be.