Technical Summary: ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity

Problem Statement

Large Language Models (LLMs) and biological AI models are rapidly acquiring capabilities relevant to biological research, ranging from literature synthesis to the interpretation of experimental data. While these advancements offer opportunities for scientific discovery, they simultaneously shift the landscape of biosecurity risks. Specifically, "AI agents"—LLMs augmented with software tools and execution environments—are increasingly capable of performing complex, end-to-end biological tasks that previously required experienced human biologists.

Current biosecurity safeguards, such as watermarking, unlearning, and nucleic acid synthesis screening, rely on the ability to confidently measure relevant AI capabilities. However, existing biology benchmarks predominantly test knowledge via short-answer or multiple-choice questions. These fail to capture the capabilities of modern agents that can autonomously use bioinformatics packages, analyze data, and write software to control laboratory automation. There is a critical need for benchmarks that measure "agentic bio-capabilities"—specifically those involving dual-use risks where AI could empower malicious actors to engineer hazardous biological entities.

Methodology

The authors introduce ABC-Bench (Agentic Bio-Capabilities Benchmark), a suite of tasks designed to evaluate AI agents on both benign and dual-use biology tasks. The benchmark is built upon seven design principles: measuring dual-use capabilities, testing AIs as agents with tool access, assessing diverse capabilities and risk chains, using objective/reproducible scoring, supporting high-throughput assessment, and including precisely specified human baselines.

Benchmark Tasks

ABC-Bench consists of three specific tasks, each corresponding to a step in a potential pathway to acquiring a hazardous DNA sequence:

1. Fragment Design: An in silico task where the agent must write Python code to design short DNA fragments. These fragments must be suitable for ordering from commercial synthesis vendors and capable of being assembled into a target sequence via Gibson Assembly.
2. Screening Evasion: A creative ideation task where the agent must design short DNA fragments that evade common nucleic acid synthesis screening (NASS) methods. The agent must obfuscate the sequence to avoid detection while ensuring the fragments can still be reconstructed into the original target sequence.
3. Liquid Handling Robot: An automation task requiring the agent to write a Python script for an OpenTrons OT-2 liquid handling robot to perform a Gibson Assembly protocol. This task assesses the agent's ability to translate biological protocols into executable code for physical hardware.

Evaluation Protocol

• AI Agents: The benchmark was used to assess eight frontier models (including Claude Sonnet 4, Claude Opus 4, Gemini 3.1 Pro Preview, GPT-5.4, Qwen3.5, and Kimi K2.5) in an agentic scaffold with access to tools like Python REPL, Bash, Bio-coding libraries, Web Search, and OpenTrons simulators.
• Human Baselines: To contextualize performance, the authors recruited 12–13 human experts (Ph.D. level in molecular/computational biology with ≥2 years of Python experience). These baseliners were given 5 hours per task and compensated $200 per task, with strict monitoring to prevent AI assistance.
• Scoring: Tasks were scored algorithmically based on pre-specified criteria (e.g., code execution success, correct reagent volumes, successful assembly). Refusal rates were also tracked.
• Wet-Lab Validation: For the Liquid Handling Robot task, a specific validation experiment was conducted using an OpenTrons Flex robot and the NEBuilder® Hi-Fi DNA Assembly kit. The model (GPT-o4-mini-high) generated scripts based on visual deck layouts and vendor instructions, which were then executed to produce physical DNA assemblies verified by whole-plasmid sequencing.

Key Results

• Superiority over Human Baselines: All tested frontier AI agents outperformed the median human expert baseline across all three tasks.
  • Liquid Handling Robot: Models like Claude Sonnet 4.6 and Gemini 3.1 Pro Preview achieved perfect scores (1.00) across all runs. The median human baseline score was 0.20 ± 0.09.
  • Fragment Design: Claude Opus 4.6 achieved a perfect score (1.00). The median human baseline was 0.33 ± 0.12.
  • Screening Evasion: Models performed weakest here, with scores ranging from 0.39 to 0.78 (Gemini 3.1 Pro Preview). The median human baseline was 0.22 ± 0.07. Notably, several models (including Claude Opus 4.6 and GPT-5.4) refused all samples for this task due to its dual-use nature.
• Performance Characteristics: Agents demonstrated high proficiency on tasks drawing on published knowledge and well-documented protocols (Fragment Design, Liquid Handling). Performance dropped on tasks requiring novel bioinformatics reasoning and creative application of principles (Screening Evasion).
• Wet-Lab Validation: In three independent experiments, GPT-o4-mini-high successfully generated functional OpenTrons scripts that executed the complete Gibson Assembly workflow. All three experiments resulted in successful DNA assembly confirmed by sequencing. The model corrected API syntax errors (e.g., labware identifiers) in a single iteration when provided with error messages.
• Refusal Rates: Refusal rates varied significantly. Screening Evasion triggered the highest refusals (up to 100% for some models), indicating safety alignment mechanisms are active. Liquid Handling Robot had minimal refusals.

Significance and Claims

The paper claims that ABC-Bench provides the first rigorous, agentic benchmark for biosecurity that moves beyond knowledge retrieval to evaluate the execution of practical, multi-step biological tasks.

• Capability Assessment: The results provide clear evidence that agentic biological capabilities are sophisticated. Frontier AI agents can match or exceed expert human performance in designing DNA fragments and controlling laboratory automation.
• Dual-Use Implications: The ability of AI agents to perform these tasks at an expert level suggests a lowering of the expertise barrier for misuse. While the models show promise for accelerating biomedical research, they also necessitate advances in governance and misuse preparedness.
• Governance Utility: ABC-Bench is designed to inform threat models, determine when safeguards should be activated, and assess the efficacy of protective measures like unlearning. The benchmark is already being utilized by major AI firms for pre-release model testing.
• Limitations: The authors acknowledge that the current benchmark is coding-centric and may under-elicited human experts who typically use dedicated design tools rather than writing scripts. Future iterations aim to cover non-coding components and a broader range of risk chain steps.

The paper concludes that while these capabilities accelerate scientific discovery, they underscore the urgency of developing appropriate safeguards, such as tiered access to dual-use capabilities and strengthened synthesis screening, to mitigate biosecurity risks.