Protocol for the development of a tool (INSPECT-IPD) to identify problematic randomised controlled trials when individual participant data are available
This paper outlines the protocol for developing INSPECT-IPD, a new tool designed to identify problematic randomised controlled trials by evaluating and implementing forensic checks on individual participant data through a combination of empirical testing and international expert consensus.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Imagine you are a chef trying to make the world's best soup. To do this, you gather recipes from hundreds of different cooks (these are Randomized Controlled Trials, or RCTs). You taste a little bit of each recipe to see if the soup works. Usually, you just read the recipe card the cook wrote down.
But what if some of those cooks lied? What if they made up the ingredients list, or even pretended they cooked the soup at all? Reading the recipe card alone can't always catch a liar.
This paper is about building a new, super-powered magnifying glass called INSPECT-IPD. Here is how it works, broken down into simple steps:
The Problem: The "Recipe Card" isn't enough
The authors already built a tool called INSPECT-SR that checks the recipe cards for red flags. But they realized that sometimes, you need to see the actual pot of soup—the raw, individual ingredients—to know for sure if the cook is telling the truth.
In the world of science, this "pot of soup" is called Individual Participant Data (IPD). It's the raw data from every single person in the study. The paper notes that when scientists can look at this raw data, they find fake or broken studies much more often (like finding 44% of fake studies vs. only 2% when they only looked at the recipe card).
The Goal: Building the "Forensic Kitchen" Tool
The team wants to build a new tool, INSPECT-IPD, to help scientists check this raw data. But they can't just guess which checks to use. They need to build a "menu" of checks that actually work.
To do this, they are running a five-step project:
Step 1: Gathering the Tools
First, they are making a giant list of every possible way to check raw data. It's like gathering every possible kitchen gadget: scales, thermometers, microscopes, and even AI detectors. They got this list from experts and previous research.
Step 2: The "Fake vs. Real" Test
Next, they are going to test these gadgets.
- They will take 15 real datasets (real soup recipes).
- They will create 15 fake datasets using an AI (like ChatGPT) to simulate how a liar might cook up fake data.
- They will ask 30 different experts to use the gadgets on these datasets.
- The Goal: To see which gadgets actually spot the fake soup and which ones are confusing or useless. They want to know: Is this check easy to understand? Does it actually catch the bad data?
Step 3: The Expert Vote (Delphi Survey)
Once they know which gadgets work, they will ask a huge group of international experts (statisticians, doctors, editors, and even patients) to vote on them.
- They will rate each check on a scale of 1 to 9 based on: "How useful is this?" and "How easy is it to understand?"
- They want to keep only the checks that almost everyone agrees are both useful and easy to use.
Step 4: The Final Menu
The experts will meet (virtually) to decide the final list.
- If a check is great at catching liars but very hard to understand, they might move it to a "specialist section" or write a better instruction manual for it.
- The result will be a draft INSPECT-IPD tool—a step-by-step guide for checking raw data.
Step 5: The "Test Drive"
Finally, they will let real-world users try the tool.
- They will ask people who write Systematic Reviews (the soup chefs) and Journal Editors (the restaurant health inspectors) to use the tool on real studies.
- They will ask: "Was this easy to use? Did it help you decide if a study was trustworthy?"
- Based on their feedback, they will tweak the tool one last time before releasing it to the public.
Important Rules of the Game
The paper is very clear about what this tool is not:
- It's not a lie detector test: It won't say "This person is a criminal." It just says, "This data looks suspicious, so be careful."
- It's not for everyone: You can only use this if you have access to the raw data (the IPD). If you only have the published paper, you have to use the older tool (INSPECT-SR).
- It's not a magic wand: The authors admit that as AI gets better at faking data, the tool will need to be updated, just like antivirus software needs updates.
The Big Picture
The authors believe that if we have a good, easy-to-use tool for checking raw data, journals might start demanding that scientists share their raw data more often. This could make science more honest, because if you know someone is going to check your "pot of soup" with a magnifying glass, you're less likely to try to fake it.
In short: This paper is the blueprint for building a new, high-tech quality control kit for scientific studies, designed to catch fake data by looking at the raw ingredients rather than just the recipe card.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.