🧬 biology

Citation language changes little after findings fail to replicate: a longitudinal analysis

A longitudinal analysis of over 15,000 citing sentences reveals that subsequent papers rarely adjust their language to reflect failed replications or mention conflicting evidence, offering readers little indication of whether original findings have been successfully replicated.

Original authors: Dongdong Guo, Jiaxuan Li

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongdong Guo, Jiaxuan Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Science is often imagined as a self-correcting machine, where new evidence automatically updates our understanding of the world. When a study is repeated and fails to produce the same result, the expectation is that the scientific community will notice, pause, and adjust how they talk about that original finding. This process relies on the language researchers use when they cite previous work. If a claim has been shown to be shaky, the sentences describing it in new papers should ideally become more cautious, or at least mention that the original result could not be confirmed. This is how knowledge is supposed to evolve: not by erasing the past, but by refining the description of what we know and what we do not.

A team of researchers set out to test whether this self-correction actually happens in the way we expect. They focused on a specific question: when a major, pre-planned attempt to repeat a famous study fails, do the authors of later papers change the words they use to describe that original finding? To find the answer, they looked at a large collection of scientific claims from psychology, economics, and social science. They identified 110 original studies that had been part of three major replication projects. For each of these studies, they gathered thousands of sentences from later papers that cited the original work. They then used advanced computer models to read these sentences and score them on a scale of certainty, measuring how definitively the authors stated the original claim. They compared the language used for findings that failed to replicate against the language used for findings that were successfully repeated.

The results revealed a striking lack of change. The researchers found that the wording used to describe failed findings remained almost identical to the wording used for successful ones. Even years after a replication attempt showed that an original result could not be reproduced, later papers continued to state those findings with the same level of confidence. The computer models detected no significant shift toward more cautious language. In fact, the difference in certainty between the two groups was so small it was statistically indistinguishable from zero. The study also looked for explicit mentions of the failure. They found that in the vast majority of cases, authors did not mention the failed replication at all. Only a tiny fraction of the sentences, roughly three to five out of every hundred, acknowledged that the original finding had been challenged or contradicted.

This pattern held true over time. The researchers tracked the language used in the eight to eleven years following the publication of the replication results. They found no gradual drift toward caution. The sentences describing the failed claims did not become softer or more qualified as time passed. The authors of the study suggest that this might be because researchers often reuse established phrases without re-reading the original evidence or checking for new, contradictory data. The original description of a finding can become a fixed part of the scientific record, passed along from paper to paper like a standard reference, regardless of whether the underlying evidence has been shaken.

The study does not claim that scientists are ignoring the evidence on purpose, but rather that the system of how findings are communicated has a kind of inertia. Even when a finding is shown to be unreliable, the way it is described in new research often remains unchanged. This has implications for how we read science and how we might build tools to help us understand it. If the language of science does not automatically reflect when a finding has failed, then readers, editors, and even artificial intelligence systems might be misled into thinking a claim is more solid than it actually is. The researchers conclude that for science to truly self-correct, we may need better ways to link original studies directly to their replication outcomes, ensuring that the story of a finding includes its failures, not just its initial success.

Technical Summary: Citation language changes little after findings fail to replicate: a longitudinal analysis

Problem Statement
The paper addresses a gap in the literature regarding how the scientific community updates the public description of findings after large-scale registered replication projects determine that original results failed to replicate. While previous studies have established that failed replications receive fewer citations or only modest declines in citation counts, it remains unknown whether the linguistic certainty of those citations changes. The authors investigate whether later papers respond to failed replications by using more cautious language (hedging) or by explicitly mentioning the conflicting evidence, or if the wording remains static despite the new evidence.

Methodology
The study employs a longitudinal analysis of 110 findings from three registered replication projects: the Reproducibility Project: Psychology, the Experimental Economics Replication Project, and the Social Sciences Replication Project.

Data Collection: The authors retrieved 15,467 dated sentences from papers citing the original studies using the Semantic Scholar Academic Graph.
Filtering: Sentences were filtered to retain only those that explicitly restate the claim ("assertion sentences"), excluding citations used merely for methods, paradigms, or background. This resulted in 8,598 sentences for analysis.
Scoring Instrument: A large language model (Qwen2.5-72B-Instruct) scored each sentence on a continuous certainty scale from 0 (fully hedged) to 1 (definitive/causal). The model was blinded to the replication status of the claim and the publication year of the citing paper.
Validation: The scoring was validated against human annotators (Cohen's $\kappa$ = 0.65 for assertion, Krippendorff's $\alpha$ = 0.60 for certainty) and cross-validated using three other distinct large language models.
Statistical Analysis: The study utilized cluster bootstrapping over claims (1,999 resamples) to compare mean certainty levels and drift rates between "refuted" (failed replication) and "robust" (successful replication) groups. It also performed a difference-in-differences analysis around the publication dates of the replication results to detect immediate shifts in language. Finally, a stratified random sample of 150 post-refutation sentences was analyzed for explicit acknowledgment of the failure.

Key Results

Stated Certainty: There was no significant difference in the stated certainty of findings that failed replication versus those that succeeded. The mean certainty difference was 0.004 (95% CI: −0.011 to 0.019).
Temporal Drift: Over an observation window of eight to eleven years, the certainty of refuted claims did not drift downward relative to robust claims. The difference in drift rates was negligible (95% CI: −0.0036 to +0.0103).
Event Response: The publication of replication results did not trigger a detectable change in the certainty of subsequent citations. The difference-in-differences estimate was +0.0056 (95% CI: −0.0205 to 0.0303), excluding any event response larger than 0.03 on the unit scale.
Acknowledgment of Failure: Explicit acknowledgment of the replication failure in post-refutation sentences was rare. Only 4.7% of sentences (7 of 150) mentioned the failure or conflicting evidence, a rate corroborated by human annotators (3.3%).
Robustness: These null results were consistent across different language models, human validation samples, and when analyzing the psychology and economics cohorts separately.

Significance and Claims
The paper concludes that in this sample (primarily psychology), citation wording provides readers with little indication of whether a finding has successfully replicated. Failed replications are rarely mentioned, even years after the results become public.

The authors argue that this lack of linguistic adjustment suggests that citation language may be "inherited" or copied from established descriptions rather than composed anew based on a reassessment of evidence. Consequently, the public record (including textbooks, reviews, and datasets used to train language models) presents replicated and non-replicated findings with similar levels of confidence.

The paper posits that this inertia is a structural feature of scholarly communication rather than a failure of individual authors to read new evidence. It suggests that current open-science reforms, which focus on the production of new findings, do not automatically update descriptions of existing findings. The authors propose that linking original studies to replication outcomes via machine-readable links could make contrary evidence more visible to authors and editors, though they do not claim this would immediately solve the problem of citation inertia. The study explicitly notes limitations, including the modest correlation between the model and human certainty scores and the heavy weighting of the sample toward psychology, acknowledging that the findings may not generalize to fields with different citation cultures or retraction protocols.

More like this