Born Flawed? Publication-Time Citation Topology Improves Retraction Early-Warning

Technical Summary: Born Flawed? Publication-Time Citation Topology Improves Retraction Early-Warning

Problem Statement
Retracted scientific articles continue to accumulate citations for years after withdrawal, often without the citing authors acknowledging the retraction. These "zombie" citations propagate flawed results into reviews, meta-analyses, and downstream studies, compromising the reliability of the scientific record. While the annual count of retractions has risen sharply, most flawed work is never formally retracted. Current detection systems rely on metadata and full-text analysis, which often only become informative after an article has been circulated and cited. There is a critical need for early-warning signals that can identify high-risk work at or shortly after publication, ideally using information available at the time of submission, before post-publication evidence accumulates.

Methodology
The study addresses whether the structural features of a paper's citation neighborhood, measurable at publication time, can improve retraction prediction beyond standard metadata.

• Data Sources: The analysis utilizes OpenAlex for scholarly works and reference lists, and the Retraction Watch database for retraction labels.
• Sample Construction: A year-matched case-control sample was drawn, consisting of 219 retracted papers and 184 non-retracted control papers. Controls were frequency-matched to retracted papers by publication year to account for temporal drifts in retraction labeling and reference conventions. The sample is restricted to works where OpenAlex exposes the full reference list.
• Feature Engineering:
  • Baseline: A set of low-cost metadata features (publication year, reference count, author/institution/country counts, title length, language).
  • Leakage-Safe Topology Features: Three structural features computed solely from the focal paper's own reference list, available at submission time:
    1. Self-citation rate: The fraction of references sharing at least one author with the focal paper.
    2. Reference-year dispersion: The standard deviation of the publication years of the references.
    3. Reference recency gap: The difference between the focal paper's publication year and the mean publication year of its references.
  • Leakage Control: A "retraction-proximity" feature (fraction of references that are themselves retracted) was computed but excluded from the main results. This feature was deemed "leaky" because OpenAlex records current retraction status, meaning it could encode retractions that occurred after the focal paper was published, rendering it unusable for genuine early-warning.
• Modeling and Evaluation: Gradient-boosted decision trees were trained to compare the metadata baseline against the metadata plus the three leakage-safe topology features. Evaluation used two protocols:
  1. 5-fold Cross-Validation: To measure separability under random splits.
  2. Out-of-Time Split: Training on earlier publication years and testing on later years to simulate real-world deployment.
  • Uncertainty was quantified using a paired percentile bootstrap with 2,000 resamples to ensure the gain between models was statistically significant.

Key Results
The addition of publication-time topology features yielded a statistically significant improvement in retraction prediction performance:

• Out-of-Time Performance: The Area Under the Curve (AUC) increased from 0.485 (metadata only) to 0.661 (metadata + topology), a gain of +0.176 (95% CI [0.099, 0.252]). Notably, the metadata-only model performed near chance levels in the out-of-time split, whereas the topology features sustained discrimination across the temporal split.
• Cross-Validated Performance: The AUC increased from 0.552 to 0.673, a gain of +0.121 (95% CI [0.073, 0.168]).
• Bootstrap Analysis: Across 2,000 bootstrap resamples, the performance gain never reached zero, confirming the robustness of the signal.
• Leakage Contrast: When the "leaky" retraction-proximity feature was included, the out-of-time gain rose to +0.308. The authors report this only to bound the potential of unconstrained models, emphasizing that the +0.176 gain represents the achievable improvement using only admissible, pre-publication data.

Key Contributions

1. Definition of Leakage-Safe Features: The paper defines a small set of reference-side structural features (self-citation rate, reference-year dispersion, and recency gap) that can be computed at submission time without relying on post-publication data.
2. Empirical Validation of Structural Signals: The study demonstrates that retraction-prone papers occupy measurably different reference neighborhoods at the time of writing. These structural signatures exist before any post-publication evidence (such as citation counts or full-text reception) accumulates.
3. Rigorous Leakage Control: The work explicitly isolates and quarantines features that encode future information (like the retraction status of references), providing a clear distinction between usable early-warning signals and non-deployable predictive artifacts.

Significance and Claims
The paper claims a modest but clear directional contribution: retraction risk leaves a structural signature in a paper's reference list that is detectable at publication time. This discovery provides a screening signal applicable at submission time, complementing existing text-based detectors that target different failure modes (e.g., fabricated text).

The authors frame the utility of this signal as triage rather than adjudication. Given the persistence of "zombie" citations, even a modest early signal that helps editors and integrity teams narrow down where to direct limited screening resources holds practical value. The authors acknowledge limitations, including the small sample size, the restriction to works with visible reference lists, and the fact that formal retraction is a noisy proxy for unreliable work (making the results a lower bound on the true signal). The contribution is presented as a reproducible direction for future work, with the analysis harness released to allow for extension on larger, field-stratified samples.