Tracking claim changes from preprint to publication across 72,644 biomedical studies using large language models

Imagine the world of biomedical research as a massive kitchen where scientists are constantly cooking up new recipes (their studies). For a long time, these recipes were only served to the public after a strict "taste-test" by a panel of expert critics (peer review). But recently, chefs have started posting their raw, untested recipes on a public bulletin board called a preprint (like bioRxiv) so everyone can see them immediately.

Some people worry that because these recipes haven't been taste-tested yet, they might be full of mistakes or exaggerations. This study decided to settle that worry by looking at 72,644 pairs of recipes: the original version posted on the bulletin board and the final, approved version served in the restaurant after the critics gave their stamp of approval.

Here is what the researchers found, using a super-smart AI assistant (like a very careful editor) to compare the two versions:

1. The Core Message Stays the Same

Think of the main claim of a study as the headline of a news story. The researchers found that for nearly 90% of the studies, the headline didn't change much.

• 40% of the time, the headline was exactly the same.
• 50% of the time, the headline got a little polish or a slight tweak, like changing "We found a cure" to "We found a promising treatment."
• Only 10% of the time did the headline get a complete makeover.

In other words, the "soul" of the research usually survives the journey from the bulletin board to the final publication intact.

2. Scientists Get More Cautious, Not More Confident

When scientists post a preprint, they are often excited and confident. After the critics review their work, they tend to dial back the excitement slightly.

• Imagine a scientist saying, "This drug is a miracle!" in the preprint.
• In the final paper, they might change it to, "This drug shows promising results."
• The study found that scientists were twice as likely to become more careful and cautious after review than they were to become more confident. They rarely made their claims sound more dramatic after the critics looked at them.

3. Time Matters, But Trends Are Improving

• The Waiting Game: If a study sat in the review queue for a very long time (the "slow lane"), it was more likely to get a major rewrite (14% chance) compared to those that got reviewed quickly (7% chance). It seems the longer the critics chew on a recipe, the more likely they are to ask for a total recipe change.
• Getting Better: Over the years (from 2019 to 2024), the need for major rewrites has dropped significantly. It's as if the chefs are getting better at writing their recipes before they even send them to the critics.

4. The "Preprint Safety Net"

The study also noticed something interesting about papers that never made it to the bulletin board. These papers were retracted (pulled from the record because they were wrong) at twice the rate of papers that did start as preprints. It's as if posting a recipe early and letting the public see it first acts as a safety check, catching errors before they become permanent.

The Bottom Line

The main takeaway is simple: Preprints are a reliable source of truth.

Just because a scientific study hasn't been "officially" published yet doesn't mean its main findings are wild guesses. The journey from the preprint bulletin board to the final publication usually involves just a little bit of polishing, not a complete rewrite. The central claims of most biomedical research are solid, even before the final critics give their approval.