Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

1. Problem Statement

Large Language Models (LLMs) have demonstrated superior performance on biological benchmarks (e.g., protocol development, troubleshooting), raising significant biosecurity concerns. Policymakers and researchers worry that LLMs could enable "novice actors" (individuals with minimal laboratory experience) to acquire dual-use biological skills, such as viral reverse genetics, potentially facilitating the creation of biological threats.

However, a critical gap exists in the literature: Do strong LLM performance on digital benchmarks translate to improved human performance in physical, wet-lab environments? Current benchmarks (like VCT or LAB-Bench) evaluate factual knowledge and short-horizon tasks in structured digital settings but fail to capture the "somatic tacit knowledge" (e.g., aseptic technique, visual assessment of cell health, manual dexterity) required for real-world execution. Furthermore, it is unclear if novices can effectively elicit expert-level capabilities from LLMs in open-ended, complex workflows.

2. Methodology

The authors conducted a pre-registered, investigator-blinded, randomized controlled trial (RCT) to empirically assess the impact of LLMs on novice performance.

• Study Design: An 8-week, single-site, parallel-group superiority trial conducted in a Biosafety Level 2 (BSL-2) laboratory in Cambridge, MA (June–August 2025).
• Participants: $N = 153$ participants with minimal prior laboratory experience (84% rated as novices on a 30-point biology experience scale). They were randomized 1:1 into two arms:
  • Internet Arm (Control, $n=76$): Access to standard internet resources (search engines, Wikipedia, YouTube, read-only forums) but blocked from LLMs and communication tools.
  • LLM Arm (Intervention, $n=77$): Access to the same internet resources plus mid-2025 frontier LLMs from Anthropic (Opus 4, Sonnet 4), OpenAI (o3, GPT-4o, GPT-5), and Google DeepMind (Gemini 2.5). Safety classifiers were disabled on these models.
• Tasks: Participants independently completed five tasks modeling a viral reverse genetics workflow:
  1. Pre-task 1: Micropipetting (prerequisite).
  2. Task 2: Mammalian cell culture (HEK293T).
  3. Task 3: Molecular cloning (plasmid assembly).
  4. Task 4: Virus production (rAAV rescue).
  5. Task 5: RNA quantification (qPCR).
  • Constraints: No human guidance, no pre-provided protocols (only high-level objectives), and participants had to identify correct reagents from a distractor-filled inventory.
• Outcomes:
  • Primary Endpoint: Successful completion of the "core reverse genetics sequence" (Tasks 2–4).
  • Secondary Endpoints: Success rates for individual tasks, time-to-completion, number of attempts, and progression through procedural milestones.
• Analysis: Pre-registered statistical analysis plan (Fisher's exact test for primary outcomes) and post-hoc Bayesian hierarchical modeling to estimate average uplift across the workflow.

3. Key Results

A. Primary Outcome: No Significant Increase in Final Completion

The study found no statistically significant difference in the primary endpoint (completion of the full reverse genetics workflow) between the two groups:

• LLM Arm: 5.2% (4/77)
• Internet Arm: 6.6% (5/76)
• Risk Ratio (RR): 0.79 (95% CI 0.24–2.62), $P = 0.759$.

This indicates that access to frontier LLMs did not enable novices to successfully complete the complex, multi-step viral synthesis workflow at a significantly higher rate than those using standard internet resources.

B. Secondary Outcomes: Modest Uplift in Specific Areas

While final completion rates were low and similar, the LLM arm showed numerically higher success rates in four of the five tasks, with one notable exception:

• Cell Culture (Task 2): The LLM arm showed a significant improvement in the Per-Protocol Set (PPS) analysis (79.7% vs. 62.5%; $P = 0.025$) and a strong trend in the Full Analysis Set (FAS) ($P = 0.059$).
• Time-to-Completion: In the Cell Culture task, the LLM arm achieved success approximately 6 days earlier than the Internet arm ($P = 0.02$).
• Other Tasks: No significant differences were found for molecular cloning, virus production, or RNA quantification in the primary analysis.

C. Bayesian Pooled Analysis: Modest Average Uplift

Due to low event rates, the authors performed a post-hoc Bayesian hierarchical analysis to estimate the "average" effect across the workflow:

• The model estimated a 1.4-fold increase (95% CrI 0.74–2.62) in success probability for a "typical" reverse genetics task under LLM assistance.
• The posterior probability of a positive effect was 85.5%, suggesting a modest but uncertain benefit.

D. Progression Through Protocols (Ordinal Regression)

A granular analysis of procedural steps revealed that LLM users were more likely to progress further through experimental protocols, even if they did not reach final success:

• LLM participants initiated tasks at higher rates and advanced through more intermediate steps across all tasks.
• Bayesian ordinal regression showed a high probability (81%–96%) that LLM users were more likely to reach advanced procedural stages compared to the Internet arm.
• Interpretation: LLMs helped users overcome early barriers (e.g., identifying reagents, understanding initial steps) but failed to bridge the gap to final successful execution for complex tasks.

E. Usage Patterns and Limitations

• Elicitation Failure: While LLM users engaged heavily (avg. 23 prompts/day, 60% image uploads), they struggled to elicit accurate outputs for complex tasks like molecular cloning (e.g., generating incorrect DNA sequences).
• Tacit Knowledge Gap: Both groups rated YouTube as more helpful than any single LLM, suggesting that visual/demonstrative knowledge (tacit skills) is better conveyed via video than text-based LLM interaction.
• Confidence Shift: Over time, LLM users' confidence in the tool's helpfulness decreased, while Internet users' belief that LLMs would have helped increased, indicating a realization of LLM limitations in practical execution.

4. Key Contributions

1. Largest Physical-World RCT in Biosecurity: This is the largest study ($N=153$) to date evaluating LLM impact on novice performance in a physical biology laboratory, moving beyond digital benchmarks.
2. Decoupling Benchmarks from Reality: The study provides empirical evidence that high performance on in silico benchmarks (knowledge retrieval) does not guarantee utility in in vivo settings (physical execution), particularly for novices lacking tacit knowledge.
3. Methodological Innovation: The use of ordinal regression and Bayesian hierarchical modeling to capture "partial progress" offers a more nuanced metric for AI safety assessments than binary success/failure metrics.
4. Biosecurity Baseline: It establishes an empirical baseline suggesting that mid-2025 LLMs provide only a modest uplift (approx. 1.4x) for novices attempting complex dual-use workflows, challenging "worst-case" scenarios where LLMs might instantly turn novices into experts.

5. Significance and Implications

• Biosecurity Policy: The results suggest that current frontier LLMs, while powerful, are not yet a "force multiplier" capable of independently enabling novices to execute complex biological threats (like viral synthesis) at high success rates. However, the modest uplift and improved progression through protocols indicate a non-zero risk that warrants monitoring.
• AI Safety Evaluation: The study highlights the critical need for physical-world validation of AI safety assessments. Relying solely on text-based benchmarks may overestimate risks (by ignoring tacit knowledge gaps) or underestimate them (by ignoring the ability of LLMs to lower entry barriers for partial execution).
• Future Directions: The findings suggest that future biosecurity risks may evolve as LLMs improve at multimodal interaction (e.g., AR/VR guidance) and as users become more proficient at "eliciting" correct outputs. The study underscores the need for adaptive, evidence-based policy rather than static risk assessments.

Conclusion: Mid-2025 LLMs did not substantially increase the rate at which novices completed complex biological workflows, but they did provide a modest performance benefit, particularly in accelerating progress through procedural steps and reducing time-to-success for specific tasks like cell culture. This reveals a significant gap between digital benchmark performance and real-world utility, emphasizing the necessity of physical-world validation for AI biosecurity.