Attitudes and Perceptions Toward the Use of Artificial Intelligence Chatbots for Peer Review in Medical Journals: A Large-Scale, International Cross-Sectional Survey

This large-scale international survey reveals that while medical journal peer reviewers are highly familiar with AI chatbots, their actual use in peer review remains limited due to significant ethical concerns and a lack of institutional training, despite a strong expressed interest in receiving guidance for future implementation.

The Gist

Imagine the world of medical research as a massive, high-stakes library where scientists write books (research papers) to teach the world how to cure diseases. Before these books go on the shelves, they must pass through a strict "quality control" checkpoint called peer review. This is where other experts read the book to make sure the facts are right, the experiments work, and the writing is clear.

Now, imagine a new, super-smart robot assistant (an AI Chatbot) has been invented that can help these experts check the books faster. It can spot typos, check if the references are real, and even find logical holes in the arguments.

This study is like a giant town hall meeting where the researchers asked the "book checkers" (the peer reviewers) what they think about letting this robot help them.

Here is the breakdown of what they found, using simple analogies:

1. The Setup: Who Did They Ask?

The researchers didn't just ask a few people; they sent out invitations to 72,851 doctors and scientists who had recently published papers. It's like sending a flyer to every chef in a major city to ask about their cooking habits. Out of that huge crowd, 1,260 reviewers stopped to fill out the survey.

2. The Current Situation: "We Know the Robot, But We Don't Use It"

• The Good News: Almost everyone knows the robot exists. 86% of the reviewers are familiar with AI chatbots, and 87% have used them for everyday things (like writing emails or summarizing news). It's like knowing how to use a smartphone; almost everyone has one in their pocket.
• The Bad News: Despite knowing the robot, 70% of them have never used it to check a medical paper. They are still doing the heavy lifting manually. It's like a chef knowing about a new food processor but still chopping every onion by hand because they aren't sure if the machine is safe to use.

3. The Training Gap: "No One Taught Us How to Drive"

The survey found that 70% of reviewers said their workplace (universities or hospitals) hasn't given them any training on how to use these AI tools for their specific job.

• The Analogy: Imagine being handed a brand-new, complex race car (the AI) and told, "Go check the books," but nobody ever taught you how to start the engine or steer.
• The Desire: However, 60% of them raised their hands and said, "Please teach us!" They are eager to learn how to use this tool properly.

4. The Big Worries: "Is the Robot Biased?"

Even though the robot is smart, the reviewers are nervous.

• The Bias Fear: 80% of reviewers are worried about algorithmic bias.
  • The Metaphor: Imagine the robot was trained mostly on books written by people from one specific country or background. If it checks a book written by someone from a different background, it might unfairly say, "This doesn't make sense," just because it's used to a different style. The reviewers are afraid the robot might be a "prejudiced judge."
• The Trust Issue: 73% are worried about whether they can actually trust the robot's answers. If the robot makes a mistake, who is responsible?

5. The Conclusion: "Not Ready for Prime Time Yet"

The main takeaway is that while the medical community is familiar with AI, they aren't ready to let it take the wheel in peer review just yet.

• The Verdict: It's like a new self-driving car that looks cool and drives well on the highway, but the drivers are still scared to let it drive in the rain or through a busy city intersection.
• The Path Forward: Before we let AI help review life-saving medical research, we need to:
  1. Teach the drivers (provide training).
  2. Fix the sensors (solve the bias and privacy issues).
  3. Write the rules (establish clear ethical guidelines).

Until those things happen, the human experts will keep doing the heavy lifting, keeping a close eye on the robot, waiting for it to prove it's truly safe and fair.

Technical Summary

1. Problem Statement

The integration of Generative Artificial Intelligence (AICs), such as Large Language Models (LLMs), into scholarly publishing is accelerating. While these tools offer potential efficiencies in identifying methodological flaws, verifying references, and enhancing language clarity, their adoption in the critical peer review process remains contentious.

• The Gap: There is a lack of empirical evidence regarding how medical journal peer reviewers perceive the utility, reliability, and ethical implications of AICs.
• The Core Issue: Despite the technological capability, concerns regarding algorithmic bias, data privacy, transparency, and research integrity have hindered widespread implementation. The study aims to quantify the current landscape of familiarity, usage, and sentiment among medical reviewers to inform future policy and training.

2. Methodology

The study employed a large-scale, international cross-sectional survey design targeting medical journal peer reviewers.

• Sampling Strategy:
  • Target Population: Peer reviewers of medical journals.
  • Recruitment Source: Corresponding author information was extracted from MEDLINE-indexed articles added to PubMed over a specific two-month window.
  • Technical Approach: An R-based script was utilized to automate the extraction of author data from PubMed.
  • Invitation Volume: 72,851 authors were invited via email. Eligibility was restricted to those who self-identified as peer reviewers.
• Instrument:
  • A 29-item survey administered via SurveyMonkey.
  • Key Metrics Assessed: Familiarity with AICs, frequency of use (general vs. peer review), perceived benefits/limitations, ethical concerns, and institutional training status.
• Data Collection Period: April 28 to June 16, 2025.
• Response Management: Two reminder emails were sent to maximize response rates.
• Final Sample: 1,260 completed surveys were analyzed.

3. Key Contributions

• First Large-Scale Quantitative Baseline: This study provides one of the most extensive datasets to date regarding the specific attitudes of medical peer reviewers toward AICs, moving beyond anecdotal evidence to statistical analysis.
• Identification of the "Usage Gap": It highlights a significant discrepancy between high general familiarity with AICs and their actual application in the peer review workflow.
• Training Deficit Analysis: The study quantifies the lack of institutional support, revealing that the majority of reviewers operate without formal guidance on AIC integration.
• Ethical Sentiment Mapping: It systematically categorizes the primary ethical barriers (specifically algorithmic bias and trust) that currently prevent broader adoption.

4. Key Results

The data revealed a complex landscape of high awareness but low operational integration:

• Familiarity vs. Application:
  • High Familiarity: 86.2% of respondents were familiar with AICs.
  • General Usage: 87.7% had used tools like ChatGPT for general purposes.
  • Peer Review Usage: Only 29.7% had used AICs for peer review, meaning 70.3% had not.
• Institutional Support & Training:
  • Lack of Training: 69.5% reported their institutions do not provide training on AIC use in peer review.
  • Demand for Training: Despite the lack of training, 60.7% expressed a strong interest in receiving such guidance.
• Perceptions and Concerns:
  • Benefits: Perceptions were mixed; while some saw value in efficiency, others were skeptical.
  • Primary Concerns:
    • Algorithmic Bias: Cited by 80.3% of respondents as a major concern.
    • Trust & Acceptance: 73.3% expressed concerns regarding user trust and the acceptance of AI-generated reviews.
  • Ethical Barriers: Persistent worries regarding data privacy and research integrity were noted as critical hurdles.

5. Significance and Implications

This study serves as a critical roadmap for the future of medical publishing:

• Policy Formulation: The findings suggest that before AICs can be broadly integrated into peer review, journals and publishers must address the "trust deficit" and establish clear ethical guidelines to mitigate algorithmic bias.
• Educational Imperative: The high demand for training (60.7%) coupled with the current lack of institutional support (69.5%) indicates an urgent need for structured educational programs to upskill the reviewer workforce.
• Future Research Direction: The study underscores that technology readiness (high familiarity) does not equate to operational readiness. Future implementations must focus on transparency mechanisms and validation protocols to ensure research integrity.
• Strategic Outlook: The authors conclude that while AICs hold transformative potential, their deployment in medical peer review must be preceded by robust governance frameworks addressing ethics, privacy, and integrity to ensure the credibility of the scientific record.