Information loss in evidence synthesis: a meta-research study of exercise intervention reporting in randomized controlled trials and systematic reviews
This meta-research study demonstrates that systematic reviews of exercise interventions suffer from significant information loss compared to their source randomized controlled trials, primarily due to incomplete original reporting and further attrition during the synthesis process, particularly regarding equipment, adherence, provider characteristics, and tailoring methods.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a secret recipe for the world's most amazing chocolate cake. You write it down in a giant, detailed notebook (the Randomized Controlled Trial, or RCT). It lists exactly how much sugar, what kind of flour, the temperature of the oven, and even the specific brand of mixer you used.
Now, imagine a food critic (the Systematic Review, or SR) who wants to summarize the best cake recipes from 130 different bakers to tell the world, "Hey, chocolate cake is great!" The critic reads your notebook and writes a short summary.
Here is the problem this paper found: The critic threw away almost all the good stuff.
The Great Recipe Leak
The researchers looked at 130 exercise "recipes" (interventions) from original trial notebooks and compared them to the summaries written in systematic reviews. They used three different checklists to see how much detail survived the trip from the notebook to the summary.
The results were a bit of a shock:
- The Original Notebooks: The original trial authors were actually doing okay. On average, they included about 66.7% of the details needed to understand the exercise (using the TIDieR checklist) and 53.2% using the exercise-specific checklist (CERT).
- The Summaries: When the reviews summarized these trials, the details vanished. The summaries only kept 16.7% of the details from both checklists.
Think of it like this: If the original recipe said, "Use 2 cups of King Arthur flour, sifted twice, at 350°F for 25 minutes," the summary just said, "Use flour and bake."
What Got Lost in the Shuffle?
The paper measured exactly what disappeared. It wasn't just random; specific, crucial details were the first to go.
- Equipment: In 98.9% of the cases where the original trial mentioned what equipment was used (like a specific treadmill or weight machine), the review left it out entirely.
- Who did it: 94.9% of the time, the reviews forgot to say who was teaching the exercise (a PhD trainer? a nurse? a volunteer?).
- Stickiness: 97.2% of the time, the reviews didn't mention if people actually stuck to the plan (adherence) or if the teachers followed the rules (fidelity).
- Safety: 87.5% of the time, any mention of bad side effects (adverse events) was dropped.
Basically, the reviews told us that an exercise worked, but they stripped away the "how," "who," "where," and "what if it hurts" parts.
Did the "New Rules" Help?
You might think, "Wait, didn't scientists release new rulebooks (TIDieR in 2014 and CERT later) to tell authors to write better?"
The paper checked this. They looked at trials published before and after these rulebooks came out.
- The Result: The paper found no clear improvement. The amount of detail in the original notebooks stayed roughly the same (around 53.1% to 53.3% completeness) whether the rulebooks existed or not.
- The Summary Problem: The data showed a higher median information loss in reviews published after the rulebooks compared to those before. However, the authors caution that this is a descriptive finding from a small sample and may be influenced by other factors like the type of exercise or clinical area, rather than proving the rulebooks made things worse.
So, the paper argues that just having a rulebook on the shelf isn't enough. Authors aren't following it, and reviewers aren't looking for it.
The "High-Quality" Myth
The researchers also checked if the "best" reviews (Cochrane reviews, which are usually the gold standard) did a better job.
- The Finding: Even the high-quality Cochrane reviews lost a massive amount of info. While they lost slightly less than the other types of reviews, they still dropped about 22.6% of the available details.
- The Takeaway: Being a "high-quality" review doesn't mean you are good at keeping the recipe details. You can be a perfect scientist but still summarize the recipe as "eat cake."
The Verdict
The paper concludes that we are losing information in two places:
- At the source: The original trial authors aren't writing down enough details to begin with.
- During the summary: Even when the details are there, the reviewers are throwing them away.
The authors suggest that if we want to know if an exercise program can be copied in a real gym or a hospital, we need both the original authors to write better recipes and the reviewers to stop summarizing them into vague labels like "aerobic exercise." Without both steps, we know the cake tastes good, but we have no idea how to bake it ourselves.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.