Faster science, penalties in evaluation, and concerns on quality and impact: Researchers' use and perceptions of preprints

While biomedical researchers actively utilize preprints for rapid dissemination and networking, their broader adoption is hindered by concerns over evaluation penalties, reliance on author reputation for credibility, and fears regarding scientific integrity, necessitating reforms in assessment frameworks and quality control mechanisms.

The Gist

Imagine the world of scientific research as a massive, high-stakes marathon. For decades, runners (scientists) have had to wait at a starting gate for a long time while officials (peer reviewers) check their shoes, verify their training logs, and stamp their passports before they are allowed to run. This "peer review" process ensures quality, but it can take a year or more.

Recently, a new rule was introduced: "Preprints." This allows runners to start running immediately, sharing their progress with the world before the officials give their final stamp. It's like posting a live blog of your race instead of waiting for the official results to be printed in the newspaper.

This paper is a big survey of runners, coaches, and race judges in the US and Canada to see how they feel about this new rule. Here is what they found, explained simply:

1. The "Speed" vs. "Dialogue" Dilemma

The Metaphor: Imagine a town square where people shout news.
The Finding: Scientists are using preprints, but mostly to shout their news first, not to have a conversation.

• Why they use them: The main reason is speed. They want to be the first to say, "I found this!" before someone else does. It's like being the first to tweet a breaking news story.
• What they don't want: Surprisingly, they don't really want people to critique their work while it's running. Only a tiny fraction of scientists use preprints to get feedback. They treat preprints like a "broadcast" (one-way street) rather than a "forum" (two-way street). They want to establish ownership of an idea quickly, not necessarily improve the idea through public debate.

2. The "Trust Me, I'm Famous" Filter

The Metaphor: Walking into a library where half the books haven't been edited yet.
The Finding: Because preprints haven't been checked by the "editors" (peer reviewers), people are nervous about whether the information is true.

• How they cope: Since there's no "quality stamp," readers look for famous names on the cover. If a famous scientist wrote it, people assume it's good. If a young, unknown researcher wrote it, people are skeptical.
• The Problem: This creates a "rich get richer" cycle. Famous scientists get read and cited; young scientists get ignored, even if their work is actually great. It's like judging a movie solely by the name of the director, ignoring the script.

3. The "Career Ceiling" (The Penalty Box)

The Metaphor: A job interview where you bring a sketch of your work instead of a finished painting.
The Finding: This is the biggest barrier. While preprints help scientists get famous fast, they can hurt their careers slowly.

• The Trap: Young scientists (junior researchers) love preprints because they need visibility. However, the "Gatekeepers" (people who hire professors or give out grants) often treat preprints as "less valuable" than published papers.
• The Penalty: If you put a preprint on your resume, a hiring committee might think, "This isn't finished yet," or "It hasn't been checked." They might give you less credit than if you had a published paper. It's like a student getting a lower grade for a draft essay than for a final one, even if the draft is brilliant.
• The Result: Many scientists are afraid to post preprints because they don't want to risk their job or promotion.

4. The "Fake News" and AI Fear

The Metaphor: A flood of water that might contain trash.
The Finding: People are worried that without the "editors" checking the work, bad information will spread.

• The Fear: In fields like medicine, if a preprint says "This drug cures cancer" but it's wrong, people could get hurt.
• The AI Threat: The paper highlights a new, scary worry: Artificial Intelligence. AI can write thousands of fake scientific papers in minutes. If preprints are just a "free-for-all," AI could flood the system with garbage, making it impossible to tell what is real science and what is a robot hallucination. This could make the public lose trust in science entirely.

The Bottom Line

The paper concludes that while preprints are a great tool for speed, the system is broken because:

1. People use them for speed, not for improving science.
2. The "Gatekeepers" (hiring committees) punish people for using them.
3. We have no good way to check if a preprint is "real" other than trusting the author's reputation.

The Solution? We need a new system. We need a way to signal that a preprint is high-quality (maybe a "quality check" label) so that hiring committees feel safe giving credit to it. Until then, the "speed" of science is being held back by the fear of "quality," and young scientists are the ones paying the price.

Technical Summary

1. Problem Statement

The scientific community seeks mechanisms to accelerate discovery, with preprints (manuscripts shared prior to peer review) proposed as a solution to reduce publication delays. While preprint adoption has surged in fields like computer science, biomedical adoption has been slower due to concerns over scientific integrity, misinformation, and healthcare risks.

Despite policy pushes from agencies like the NIH, a critical gap exists in understanding the multifaceted perceptions of preprints across different academic roles (authors, readers, and evaluators). Existing literature lacks large-scale empirical evidence on:

• The interplay between preprint usage behaviors (reading, citing, submitting) and underlying motivations.
• Perceptions of preprint quality and their impact on career advancement.
• How institutional gatekeepers (grant reviewers, hiring committees) credit preprints in formal evaluations.
• The specific anxieties regarding AI-generated content and declining public trust in science.

2. Methodology

The study employs a large-scale mixed-methods survey targeting biomedical scholars in the United States and Canada.

• Data Collection:

  • Sampling: Researchers extracted email addresses from 13 million publication records (iCite/NIH Open Citation Collection) matched with Web of Science, filtering for US/Canada institutions and publications since 2017.
  • Sample Size: 281,940 unique emails were invited. The survey ran from March to May 2025.
  • Response: 5,855 initiations (2.7% response rate); 1,874 completions. After cleaning invalid responses, the final analytical sample was 1,758 participants.
  • Demographics: 83.7% US-based, 16.3% Canadian; 76.7% from higher education/research; 53.3% full professors, with significant representation of junior scholars (assistant professors, postdocs, PhD students).

• Analytical Approach:

  • Quantitative: Ordinal logistic regressions were used to examine group differences (by rank, gender, country, discipline) in usage frequency, motivations, perceived quality, and evaluation credit.
  • Qualitative: Deductive content analysis with sentiment classification was applied to 424 free-text comments, categorized into three themes: perceived quality, impact on community/society, and impact on professional development.

3. Key Contributions

This paper provides the first comprehensive, large-scale empirical analysis of preprint ecosystems in biomedicine that integrates usage behaviors with evaluation perceptions. It moves beyond simple adoption metrics to reveal the structural tensions between the speed of preprints and the rigor required for academic credit.

4. Key Results

A. Usage Patterns and Motivations ("Speed over Dialogue")

• High Engagement: 56.6% of respondents read 1–5 preprints/month; 43.3% submitted 1–5 preprints in the last two years.
• Primary Driver: The dominant motivation is timeliness (46.6% for reading, 70.0% for citing).
• Lack of Dialogue: "Receiving open peer review" was the lowest priority for submission (74.4% rated it unimportant). Researchers use preprints to broadcast results and establish priority, not to invite critique.
• Demographic Variance: Junior scholars are significantly more active in reading, citing, and submitting than senior scholars.

B. Quality Perception and Heuristics

• Quality Concerns: Uncertainty about quality is the primary deterrent for non-users (66.8% of non-readers).
• Reputation Filtering: In the absence of peer review, scholars rely heavily on author reputation as a heuristic. 46.0% of regular users consider familiar author names a critical factor for reading/citing.
• Transparency Paradox: While scientific design is rated highly, material availability (data/code) is a major weakness; over 33% hold negative views on the availability of research materials in preprints.
• Demographic Bias: Women are more likely to rely on author lists as a filter than men.

C. The "Paper Ceiling" and Evaluation Penalties

• Soft Benefits vs. Hard Penalties: Preprints are viewed positively for networking, citations, and collaboration (especially by junior scholars). However, they are perceived as detrimental to career advancement (tenure/promotion).
• Gatekeeper Bias:
  • Grant Review: ~50% of reviewers assign less credit to preprints than peer-reviewed publications.
  • Hiring/Promotion: Over 55% of committee members assign less credit to preprints in job applications and tenure reviews.
  • Junior Disadvantage: Junior scholars are more likely to value preprints but face the highest risk of career penalties due to the conservative stance of senior gatekeepers.
• Reason for Non-Submission: "Lack of recognition in research evaluation" is the leading reason scholars avoid submitting preprints, outweighing fears of idea scooping.

D. Societal Impact and Emerging Threats

• Positive Views: Preprints are credited with accelerating knowledge sharing and reducing reliance on paywalls.
• Negative Views: The primary concerns are misinformation, fraud, and AI threats.
  • 42.3% worry about misleading information.
  • 38.0% fear AI-generated papers will flood the ecosystem with low-quality or fraudulent content, exacerbating the "Matthew Effect" (where established labs gain disproportionate visibility).
• Trust Erosion: There is significant concern that unvetted preprints, especially those amplified by social media or AI, will further erode public trust in science.

5. Significance and Implications

The study concludes that the preprint ecosystem in biomedicine is currently stuck in a "speed vs. rigor" dilemma. While preprints successfully accelerate dissemination, their potential is stifled by:

1. Evaluation Systems: Academic promotion and funding structures remain conservative, penalizing preprint usage and reinforcing the status quo.
2. Quality Control: The lack of formal quality signaling forces reliance on author reputation, which disadvantages early-career researchers and reinforces inequality.
3. AI Risks: The potential for AI to generate fraudulent preprints threatens to undermine the credibility of the entire system.

Recommendations:

• Reform Evaluation: Institutions must adapt tenure and grant review frameworks to formally recognize preprints as valid research outputs.
• New Quality Mechanisms: There is an urgent need for scalable, non-peer-review quality signaling mechanisms (e.g., transparency badges for data/code availability) to reduce reliance on author reputation.
• AI Governance: The ecosystem must develop safeguards to detect and manage AI-generated content to maintain public trust.

Ultimately, for preprints to truly accelerate science, the ecosystem must evolve from a tool for "broadcasting" to one that fosters genuine academic dialogue while ensuring rigorous quality control.