Research Artifacts in Secondary Studies: A Systematic Mapping in Software Engineering
This systematic mapping of 537 software engineering secondary studies reveals that while the reporting of research artifacts has significantly improved over the last decade, the majority still lack them or permanent storage, prompting a call for mandatory publication to enhance transparency and reproducibility.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef who loves to write cookbooks. But instead of just writing down recipes, you write "Cookbook Reviews." You read hundreds of other cookbooks, taste their dishes, and then write a big summary telling everyone which recipes are the best, which ingredients are trending, and what the future of cooking looks like.
In the world of software engineering, these "Cookbook Reviews" are called Systematic Reviews (SRs). They are the gold standard for summarizing what we know about building software.
This paper is like a detective story where the authors went on a mission to check if these "Cookbook Reviews" are actually leaving behind the ingredients and tools (called Research Artifacts) they used to write their reviews.
Here is the breakdown of their investigation, using some everyday analogies:
1. The Big Problem: The "Magic Recipe" Mystery
When a chef writes a review, they should ideally say, "Here is the exact list of ingredients I used, and here is the pot I cooked in, so you can try it yourself." In software, this is called a Research Artifact. It could be the code, the data, or the list of papers they analyzed.
The authors of this paper asked: "Are these software reviewers actually sharing their ingredients?"
2. The Investigation (The Method)
The team acted like librarians. They went through 537 different "Cookbook Reviews" published between 2013 and 2023. They looked for clues:
- Did the author say, "Here is my data"?
- Did they put the data in a permanent locker (like a digital vault called Zenodo or Figshare) that has a permanent ID card (a DOI)?
- Or did they just say, "Ask me for it," or put a link to a personal website that might disappear in a year?
3. The Findings: A Mixed Bag
Here is what they found, translated into plain English:
- The Good News: Things are getting better! In 2013, almost no one shared their ingredients. By 2023, about 62% of the reviewers were sharing something. It's like a slow but steady trend where more chefs are finally saying, "Hey, here's my recipe!"
- The Bad News: Even in 2023, 38% of the reviewers still kept their ingredients hidden. They didn't share the data at all.
- The "Leaky Bucket" Problem: Of the ones who did share, many didn't use a "permanent locker." They used personal websites or email requests.
- Analogy: Imagine a chef saying, "My recipe is on my kitchen counter." But in five years, the chef moves, the house is sold, and the recipe is gone. That is what happens with non-permanent links. They found that even in 2023, some links were already "dead" (broken).
- The "Permanent Vault" Stat: Only about 30% of the 2023 studies used a permanent, unbreakable digital vault (with a DOI). This is the ideal way to store things so they never disappear.
4. Why Does This Matter? (The Four Reasons)
The authors explain why sharing these "ingredients" is crucial:
- Trust: If you can't see the ingredients, how do you know the chef didn't just make up the taste? Sharing the data proves the review is honest.
- Re-doing the Work (Replication): Science is about being able to do the same thing twice and get the same result. If you can't see the data, you can't check if the review is right.
- Updating the Menu: Software changes fast. If a review is 5 years old, it might be outdated. If the original data is saved in a vault, future chefs can easily update the review with new ingredients.
- Robots Helping Out: Writing these reviews takes forever (about a year per review!). If the data is saved nicely, we can eventually use AI robots to help write these reviews. But robots can't work if the data is hidden or broken.
5. The Verdict and The Future
The authors conclude that while we are moving in the right direction, we aren't there yet.
- The Recommendation: They are asking all software journals to make it mandatory for reviewers to drop their "ingredients" into a permanent digital vault.
- The Goal: They want to reach a point where 100% of reviews include these artifacts, and they are all stored in a way that ensures they will be accessible 10, 20, or 50 years from now.
In a nutshell: The paper is a call to action for software researchers to stop hiding their work. They want everyone to be transparent, so that science can be trusted, checked, and built upon, rather than disappearing into the digital void.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.