📊 statistics

Does Open Peer Review Correlate With Better Altmetric Attention Performance?An Empirical Analysis Based on Nature Communications

Based on an empirical analysis of 35,852 papers from *Nature Communications* using propensity score matching, this study demonstrates that Open Peer Review significantly correlates with higher altmetric attention scores, suggesting that transparent review processes enhance the social impact and public dissemination of scientific research.

Original authors: Xiaotian WEI, Zhengyi GUAN, Chenyang WANG, Jianxun CHU

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaotian WEI, Zhengyi GUAN, Chenyang WANG, Jianxun CHU

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of science, a paper's journey does not end when it is printed. Once published, research enters a vast, noisy marketplace of attention, where it competes for eyes on social media, mentions in news articles, and discussions in policy documents. To measure this kind of public reach, researchers have developed a new set of tools called altmetrics. Unlike traditional citations, which count how often other scientists reference a paper in their own work, altmetrics track how widely a study spreads through the broader world. These scores act as a thermometer for social impact, capturing everything from a tweet to a government report. At the same time, the way scientists evaluate new research is also changing. For decades, the standard process has been a closed room where anonymous experts critique a manuscript before it sees the light of day. A newer approach, known as open peer review, invites transparency by allowing the public to see the reviewers' names and their written reports. This shift aims to build trust and accountability, but it raises a practical question: does making the review process open actually help a paper reach more people, or does it simply remain a technical detail for specialists?

A team of researchers set out to answer this question by looking at the digital footprint of thousands of scientific papers. They focused their investigation on Nature Communications, a major journal that has allowed authors to choose whether to make their review process public since 2016. This unique setup created a natural experiment. For several years, the journal published two types of papers side by side: those where the review history was hidden and those where it was open. The researchers gathered data on 35,852 research articles published between 2016 and 2022. They wanted to know if the papers with open reviews consistently attracted more attention from the public and the media than those with closed reviews, after accounting for other factors that might influence popularity, such as the topic of the study, the number of authors, or when the paper was published.

To ensure a fair comparison, the researchers used a statistical method that pairs each open-review paper with a closed-review paper that is nearly identical in every other way. Imagine lining up two groups of runners who are the same age, height, and training level, but one group wears a special pair of shoes while the other does not; this method helps isolate the effect of the shoes alone. By matching the papers this way, the team could see if the open review itself made a difference. They found that papers with open peer reviews did indeed perform better in the court of public opinion. On average, these papers received a higher attention score, a difference that remained significant even after the researchers controlled for the paper's subject, length, and the number of countries involved. The study calculated that the open review process added a measurable boost to a paper's visibility, suggesting that transparency acts as a bridge between the laboratory and the living room.

The researchers explored why this might be happening. One possibility is that journalists and science communicators, who act as gatekeepers for the public, feel more confident covering research when they can see the review process. When a reporter can read the actual comments and see who vouched for the science, the story feels more credible and easier to trust. This transparency may encourage media outlets to pick up the story and share it with a wider audience. Another factor could be that the open review process encourages authors to write more clearly. Knowing that their critics' names and words will be public, reviewers might offer more constructive feedback, and authors might refine their writing to be more accessible to non-experts. This clarity could make the final paper easier for the general public to understand and share.

However, the study also offered important nuances to this positive finding. The researchers noted that having an open review is not a magic ticket to viral fame. In their dataset, the single most popular paper, which garnered nearly 20,000 points of attention, did not use open peer review. This suggests that while open reviews help on average, a paper's ultimate reach still depends heavily on its content and the nature of the topic. Furthermore, the researchers pointed out that a high attention score does not always mean the public loves the science. Sometimes, a high score comes from controversy or heated debate. An open review might expose scientific disagreements, drawing in critics and sparking arguments that drive up the numbers just as much as praise would. The score measures the volume of attention, not necessarily the quality of the reception.

The authors concluded that while open peer review is a powerful tool for expanding the reach of science, it should be viewed as part of a larger ecosystem of communication. For journal editors, the findings suggest that encouraging open reviews could be a strategic way to connect with the public, provided they also help authors explain their work in plain language. For those who evaluate scientific success, the study serves as a reminder that social media attention is a complex metric. It reflects a mix of genuine interest, curiosity, and controversy. The research confirms that transparency in the review process correlates with greater public engagement, offering a new path for science to step out of the ivory tower and into the public conversation, but it does so with a clear understanding of the limits and the realities of how people consume information.

Technical Summary: Open Peer Review and Altmetric Attention Performance

Problem Statement
While Open Peer Review (OPR) is a cornerstone of the open science movement, empirical consensus regarding its impact on academic influence remains elusive. Existing literature has largely focused on traditional citation metrics, yielding inconsistent results; some studies suggest OPR increases citations, while others find no significant correlation or even a negative effect. Crucially, the relationship between OPR and altmetrics—indicators measuring the social dissemination and attention of research beyond the academic community—has not been systematically examined. This study addresses the gap in understanding whether the transparency mechanisms of OPR (such as public review reports and disclosed reviewer identities) correlate with enhanced social visibility and attention, as measured by the Altmetric Attention Score (AAS).

Methodology
The study employs a quasi-experimental design using Propensity Score Matching (PSM) to mitigate selection bias inherent in non-randomized adoption of OPR.

Data Source: The sample consists of 35,852 research articles published in Nature Communications between 2016 and 2022. This journal was selected because it implemented a transparent review system in 2016 where authors could choose to make review reports public, creating a natural coexistence of OPR and non-OPR papers within a unified institutional framework.
Variable Definition:
- Treatment Variable: A binary dummy variable indicating whether a paper adopted OPR. OPR was defined comprehensively to include papers with disclosed reviewer identities, public review reports, or both. Identification was performed via Python scraping and verified by Large Language Models (DeepSeek V3.2) and manual cross-checking.
- Outcome Variable: The Altmetric Attention Score (AAS), a weighted metric aggregating attention from social media, news, policy documents, and blogs.
- Confounding Variables: Twelve covariates were controlled for, including title length, abstract word count, number of pages, reference count, author count, number of countries involved, cross-national collaboration status, disciplinary field (categorized into four major groups), publication date, and article access counts.
Analytical Strategy:
1. Logistic Regression: A logit model was used to estimate the propensity score (probability of adopting OPR) for each paper based on the 12 covariates.
2. Matching: Radius matching (caliper = 0.01) was applied to pair OPR papers with non-OPR papers having similar propensity scores. Balance was verified by ensuring standardized biases dropped below 5% and variance ratios approached 1.
3. Effect Estimation: The Average Treatment Effect on the Treated (ATT), Average Treatment Effect on the Untreated (ATU), and Average Treatment Effect (ATE) were calculated.
4. Robustness Checks: The study employed five alternative matching algorithms (Kernel, Local Linear Regression, K-Nearest Neighbors with k=1 and k=2, and Spline Matching), sensitivity analysis using Rosenbaum bounds to test for unobserved confounding, and truncated sample analysis (excluding the top 10% of AAS scores) to address long-tail distribution issues. A supplementary analysis used tweet counts as an alternative outcome variable.

Key Results

Primary Finding: After controlling for confounding factors, papers adopting OPR demonstrated a significantly higher Altmetric Attention Score compared to non-OPR papers. The Average Treatment Effect on the Treated (ATT) was 5.411 (95% CI: [1.34, 9.48]).
Robustness: The positive correlation held across all five alternative matching methods, with ATT estimates ranging from 4.792 to 6.786, all statistically significant. Sensitivity analysis indicated the results were robust even under the assumption of substantial unobserved confounding (up to $\gamma = 3$ ).
Distributional Nuance: While the mean difference decreased when excluding the top 10% of high-scoring papers (ATT dropped to 1.157), the effect remained positive and significant. This suggests OPR benefits the broader population of papers, though the effect is more pronounced for papers already attracting high attention.
Specific Channels: The positive association was confirmed using tweet counts as a standalone metric, indicating the effect is not merely an artifact of the composite AAS weighting scheme.
Temporal Stability: Interaction analysis revealed that the positive effect of OPR did not vary significantly by publication year, suggesting the finding is stable over time and not driven by temporal trends in social media usage.

Significance and Claims
The paper claims to provide empirical evidence that OPR is significantly and positively correlated with the expanded dissemination scope and social influence of academic outputs. The authors argue that this correlation likely stems from two mechanisms:

Transparency and Trust: Public review histories provide non-specialist audiences (including journalists and policymakers) with credible contextual information about the scientific validation process, potentially lowering barriers to understanding and fostering trust.
Quality and Readability: The accountability of OPR may incentivize reviewers to provide more constructive, detailed feedback, leading to manuscripts that are more readable and accessible to the public.

The study concludes that implementing transparent peer review systems can serve as a strategic pathway for journals to connect the academic community with broader social audiences. However, the authors explicitly caution that a higher Altmetric Attention Score reflects attention rather than endorsement, noting that high scores can also stem from controversy or critical debate. Therefore, while OPR enhances visibility, it does not guarantee positive public sentiment. The findings are presented as a robust correlation rather than a definitive causal proof, acknowledging potential self-selection bias where authors of high-potential social impact papers may be more likely to choose OPR.

Limitations Acknowledged
The study acknowledges that its findings are specific to Nature Communications and may not generalize to specialized journals or disciplines with different communication norms. Furthermore, the inability to fully control for unobserved author behaviors (such as active social media promotion by authors who choose OPR) remains a limitation, though Rosenbaum bounds analysis suggests the results are resilient to such unobserved confounding.

Technical Summary: Open Peer Review and Altmetric Attention Performance

More like this