Does the type of publisher response to integrity concerns influence subsequent citations? A cohort study.

This cohort study found that the type of editorial response (retraction, expression of concern, or editorial notice) issued for integrity concerns in randomized controlled trials did not significantly influence subsequent citation rates or accelerate citation decline compared to control trials.

The Gist

Imagine a library where scientists write books (research papers) to share their discoveries. Sometimes, years later, someone realizes a book contains serious errors or even lies. The library's managers (the journal publishers) have a few ways to handle this:

1. The "Red X" (Retraction): They slap a big "RETRACTED" stamp on the cover and say, "This book is garbage, throw it away."
2. The "Warning Label" (Expression of Concern): They put a sticky note on the cover saying, "Hey, there are some serious doubts about this, be careful."
3. The "Small Note" (Editorial Notice): They write a tiny, barely visible note in the back of the book saying, "We are looking into some concerns."

The Big Question:
The authors of this study wanted to know: Does putting these stamps, notes, or warnings actually stop people from borrowing and quoting these books?

They wondered if a big "Red X" would make people stop reading the book immediately, or if a tiny note would be ignored completely. They also wondered if people just keep quoting the books anyway, even after the warnings are posted.

The Experiment: A Race Against Time

The researchers gathered a group of 172 specific "problem books" (scientific studies) that had received one of these three types of warnings. To see if the warnings worked, they compared these books to a group of "normal books" published in the same journals at the same time that had no problems.

They tracked how many times these books were quoted (cited) by other scientists over several years, looking at the timeline before and after the warnings were posted.

The Surprising Results

Here is what they found, using a simple metaphor:

The "Natural Sunset" Analogy
Imagine a sunset. When a new scientific paper comes out, it's like the sun rising. Everyone is excited, and citations (people quoting the paper) go up and up. Eventually, the sun reaches its highest point (the peak), and then it naturally starts to set. The paper gets older, newer research comes out, and fewer people quote it. This is the "natural decline."

The Finding:
The researchers found that when a warning (Retraction, Warning Label, or Small Note) was posted, the books didn't stop getting quoted any faster than the "natural sunset."

• Did the "Red X" work? No. The books with retractions didn't stop being quoted faster than the normal books that were just getting old.
• Did the "Warning Label" work? No. The books with "Expressions of Concern" didn't stop being quoted faster either.
• Did the "Small Note" work? Definitely not. The tiny notes were so invisible that they had zero effect.

In fact, the rate at which these "problem books" stopped being quoted was exactly the same as the rate at which the "normal books" stopped being quoted. The warnings didn't speed up the process at all.

Why Does This Happen?

The study suggests a few reasons why the warnings fail to stop the citations:

1. The "Too Late" Problem: By the time the publisher finally puts the warning on the book, the book has usually already been quoted thousands of times. The "sunset" has already started. The warning arrives just as the book was naturally fading away anyway.
2. The "Invisible Ink" Problem: Some warnings (like the "Small Note" or Editorial Notice) are hidden on the publisher's website. If a scientist is using a reference manager or searching on Google, they might never see the warning. They just see the book title and quote it.
3. The "Copy-Paste" Habit: Scientists often copy references from other papers. If Paper A quoted the "problem book" before the warning came out, and Paper B copies Paper A's reference list, Paper B will accidentally quote the bad book too, even if they never looked at the original warning.

The Takeaway

This study is a bit of a wake-up call for the scientific world. It suggests that publishing a warning after the fact is often too little, too late.

If a journal wants to stop bad science from spreading, they can't just wait until years later to slap a "Warning" sticker on it. The current system of "Expressions of Concern" and "Editorial Notices" isn't doing enough to stop people from citing unreliable research. The "Red X" (Retraction) is the strongest signal, but even that doesn't seem to stop the citations faster than time does on its own.

In short: The library managers are trying to stop people from borrowing broken books by putting up signs, but the borrowers are either ignoring the signs, can't find them, or have already checked the books out so many times that the signs don't matter anymore.

Technical Summary

1. Problem Statement

Research integrity concerns often lead to editorial responses from journals, ranging from Editorial Notices (often less visible, not indexed in major databases) to Expressions of Concern (EoC) and Retractions (highly visible, indexed). While it is assumed that these responses signal unreliability and should reduce future citations, the actual impact of different response types on citation trajectories remains unclear.

• The Gap: Existing literature is mixed regarding retractions (some show citation decline, others show persistence or even increases). Crucially, there is a lack of research comparing the effectiveness of Editorial Notices and EoCs against Retractions and against control groups (unaffected papers) to determine if the type of notice influences the rate of citation decline.
• Hypothesis: The authors hypothesized that more visible and serious responses (Retractions, EoCs) would cause a steeper decline in citations compared to less visible notices (Editorial Notices) and compared to the natural decline seen in control papers.

2. Methodology

The study employed a retrospective cohort design using OpenCitations data.

• Study Population (Index Set):
  • Source: 172 Randomised Controlled Trials (RCTs) published by a single research group between September 2011 and January 2019.
  • Integrity Status: All 172 trials had integrity concerns.
  • Intervention Groups:
    • Editorial Notices: 41 papers.
    • Expressions of Concern (EoC): 38 papers.
    • Retractions: 23 papers.
    • No Action: 70 papers (served as an internal index control).
• Control Group:
  • Selection: RCTs published in the same journal and same year as the index papers, excluding the senior author of the index set and any papers already flagged in the Retraction Watch database.
  • Scale: 3,330 control trials identified for 151 of the 172 index trials.
  • Refinement: A "highly cited" control subgroup ($n=1,598$) was created (papers with $\ge$25 citations) to match the citation volume of the index trials, as the initial control group had significantly lower citation counts.
• Data Collection:
  • Source: OpenCitations database (via Scopus, Web of Science, PubMed).
  • Timeframe: Citations collected up to 2025 (covering ~60 months before and after the editorial response).
  • Metrics: Monthly citation rates per paper.
• Statistical Analysis:
  • Trend Analysis: Calculated the slope of the line of best fit for citations 60 months before and after the "event" (editorial response for index papers; peak citation month for controls/no-action papers).
  • Comparisons:
    • Compared slopes between different editorial response types (Notice vs. EoC vs. Retraction).
    • Compared slopes of index groups against control groups.
    • Used linear regression models to test interactions between time and category.
    • Used Wilcoxon tests for individual paper slope changes and t-tests for group differences in slope differences.
  • Software: R 4.4.2.

3. Key Contributions

• Comparative Analysis of Notice Types: This is one of the first studies to directly compare the citation impact of Editorial Notices (a newer, less visible mechanism) against EoCs and Retractions.
• Controlled Baseline: By using contemporaneous control trials from the same journals and years, the study isolates the effect of the integrity notice from natural citation decay patterns.
• Methodological Transparency: The study utilizes open-source tools (OpenCitations, R) to demonstrate that sophisticated meta-research can be conducted without expensive commercial database access.

4. Results

• Citation Trajectories:
  • Both index and control papers showed a steady increase in citations, peaking 45–65 months after publication, followed by a natural decline.
  • The median peak citation month for index trials was very close to the median date of the editorial response, suggesting many responses were issued after the citation peak had already occurred.
• Impact of Editorial Response Type:
  • No Significant Difference: There were no statistically significant differences in the rate of citation decline between papers receiving a Retraction, an EoC, or an Editorial Notice.
  • Decline Rates: Post-response, the monthly citation decline rates were similar across all groups (approx. 0.0098 to 0.012 citations/month per month).
• Comparison with Controls:
  • The rate of citation decline for all index groups (regardless of response type) was not statistically different from the natural decline observed in the highly cited control group (0.0099 decline rate).
  • Even the "No Action" index group showed a decline rate (0.014) similar to the other groups.
• Individual Paper Analysis:
  • Analysis of individual paper slopes before and after the event confirmed no pairwise significant differences in the change of slopes among the groups.

5. Significance and Implications

• Ineffectiveness of Current Notices: The study concludes that editorial responses (Retractions, EoCs, Notices) do not meaningfully accelerate the decline in citations beyond the natural lifecycle of a paper.
• Timing is Critical: The authors suggest that the lack of impact may be due to latency. Since citations typically peak 4–5 years post-publication, and editorial responses in this dataset were issued on average ~5 years post-publication, the damage (citations) may have already been done. The decline observed was likely the natural post-peak decay, not a reaction to the notice.
• Visibility Issues: The finding that "less visible" notices performed similarly to "highly visible" retractions suggests that visibility alone is insufficient to stop citations. Authors may be unaware of notices due to:
  • Lack of indexing in reference managers.
  • Failure to check primary sources.
  • Copying references from secondary sources that haven't been updated.
• Policy Implications:
  • The current system of issuing editorial responses may provide little value in preventing the continued citation of unreliable research if issued late.
  • Future guidelines (e.g., NISO) must focus on timeliness (issuing notices earlier) and systemic integration (ensuring retraction/notice data is automatically updated in all reference management software and databases) to be effective.

Conclusion: The publication of an editorial response, regardless of its severity or visibility, did not alter the citation trajectory of these RCTs more than the natural decline seen in comparable, uncontaminated literature. This raises serious questions about the efficacy of current publisher integrity protocols in mitigating the spread of unreliable science.