Code Sharing In Prediction Model Research: A Scoping Review
This scoping review of nearly 4,000 prediction model studies reveals that while code sharing is increasing, it remains uncommon and often lacks the documentation, dependency specification, and modular structure necessary for true reproducibility, thereby providing an empirical foundation for developing the TRIPOD-Code reporting guideline.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef who just invented a revolutionary new recipe for a soup that cures the common cold. You write a paper about it, describing the ingredients and the steps. But you don't share the actual recipe card or the list of specific spices you used.
Now, imagine another chef tries to make the soup. Without your exact recipe, they guess the amounts of salt, use a different brand of broth, and skip a step. The result? A soup that tastes nothing like yours. In the world of medical research, this is exactly what happens with prediction models.
This paper is a massive investigation into how scientists are sharing the "recipe cards" (the computer code) for their medical prediction models. Here is the story of what they found, explained simply.
The Big Problem: The "Black Box" Kitchen
In medical research, scientists build complex computer models to predict things like: Will this patient get sick? How likely is this surgery to succeed?
To build these models, they write computer code. This code is the recipe.
- The Goal: If another scientist wants to check your work or use your model to help patients, they need your recipe.
- The Reality: Most scientists keep their recipes in a locked drawer. Even when they say, "Yes, I have a recipe," they often just hand over a crumpled napkin with a few notes, not the full, organized cookbook.
The Investigation: A Taste Test of 4,000 Papers
The authors of this paper decided to take a "scoping review" (a giant taste test) of nearly 4,000 medical studies that claimed to follow a standard set of rules called TRIPOD (a checklist for writing good medical papers).
They used a super-smart AI assistant (like a very fast, very hungry robot librarian) to:
- Read thousands of papers to see if the authors shared their code.
- Download the code if it was found.
- Inspect the code to see if it was actually usable.
What They Found: The "Recipe" Crisis
1. Most Chefs Keep Their Secrets
Out of nearly 4,000 studies, only 12% shared their code. That's like 100 chefs entering a competition, and only 12 handing over their recipe cards. While this number is slowly growing (more chefs are sharing now than in 2015), it's still shockingly low.
2. The "Shared" Recipes Are Often Broken
Of the few chefs who did share their code, the quality was often terrible.
- The README File: This is like the "Start Here" sign on a recipe box. 80% of the shared code had one, but only half of those actually explained what the code did or how to use it.
- Missing Ingredients: Many shared code didn't list the specific "ingredients" (software versions) needed. It's like a recipe saying "add flour" but not specifying if you need wheat flour, almond flour, or gluten-free flour. Without this, the soup turns out wrong.
- No License: Many didn't say if you were allowed to copy the recipe or sell it.
- No "Test Kitchen": Very few included "test scripts" (like a taste-test step) to prove the code actually works.
3. The "AI" vs. The "Human" Factor
The study found that journals that explicitly asked for code (like Nature Communications or PLOS Digital Medicine) had much higher sharing rates. It's like a cooking show that says, "If you don't show your recipe, you can't compete." When the rules are clear, chefs share more.
Interestingly, studies that cited a newer, AI-focused version of the rules (TRIPOD+AI) shared code much more often than those citing the older rules. This suggests that when we tell scientists, "Hey, AI code sharing is important," they actually do it.
The Analogy: The "Archival Box" vs. The "Cookbook"
The authors realized that most shared code is treated like an archival box (a dusty box in a basement) rather than a cookbook (a tool meant to be used).
- Archival Box: "Here is the code we used 5 years ago. Good luck figuring it out."
- Cookbook: "Here is the code, here is the list of ingredients, here is the step-by-step guide, and here is a sample dish you can try right now."
The study found that most shared code is in the "Archival Box" category. It exists, but it's not reusable.
Why Does This Matter?
If we can't reproduce the results (make the soup again), we can't trust the medicine.
- Patient Safety: If a doctor uses a model to decide on a treatment, they need to know that model works exactly as the scientist said it does.
- Wasted Money: Other scientists spend years trying to figure out how to run the code, only to fail because the "recipe" was incomplete.
The Solution: "TRIPOD-Code"
The authors are proposing a new set of rules called TRIPOD-Code. Think of this as a new, stricter "Cookbook Standard."
Instead of just saying, "Did you share the code?" (Yes/No), the new rules will ask:
- "Did you list the ingredients?" (Dependencies)
- "Is there a 'Start Here' sign?" (Documentation)
- "Is there a license?" (Legal permission)
- "Can I actually run this?" (Reproducibility)
The Bottom Line
The paper concludes that sharing code is not enough; sharing good code is what matters.
Right now, the medical research world is like a kitchen where everyone is shouting, "I have a recipe!" but no one is actually handing over the ingredients list or the instructions. The authors hope that by setting clear, simple standards for how to share code, we can turn those dusty "Archival Boxes" into open, usable "Cookbooks" that help doctors save more lives.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.