Evaluating AI-Assisted Customer Verification for Synthetic Nucleic Acid Screening

This study demonstrates that AI-assisted verification, specifically using the Gemini 2.5 Pro model integrated with external APIs, can achieve accuracy comparable to human experts in screening synthetic nucleic acid orders while reducing costs by up to 50-fold, supporting a hybrid approach where AI handles information gathering and humans retain final decision-making authority.

The Gist

Imagine you run a high-tech bakery that sells a very special ingredient: synthetic DNA. This ingredient is like a "digital recipe" for building living things. Most of the time, people buy it to make life-saving vaccines or new medicines (the good stuff). But, a bad actor could use the same recipe to cook up a dangerous virus (the bad stuff).

To stop the bad actors, the bakery has a security guard at the door. This guard's job is Legitimacy Screening: checking who is ordering the DNA and making sure they have a good reason for it.

The Problem: The Security Guard is Too Slow and Expensive

Right now, this security check is done by a human expert. It's like hiring a private detective to look up every customer's background, check their email, see if they work for a university, and search for their past research papers.

• The Catch: It takes a long time (about 15 minutes per person) and costs a lot of money (about $14 per check).
• The Result: Because it's so expensive and slow, many bakeries skip this step, leaving the door wide open for dangerous orders.

The Experiment: Can a Robot Do the Detective Work?

The authors of this paper asked: What if we use a super-smart AI (a Large Language Model) to do the detective work?

They set up a test with 5 different AI models (like different types of super-intelligent robots) and compared them against 2 human experts. They gave them 41 fake customer profiles to check. Some customers were real scientists, some worked for companies, and some were on "sanctions lists" (like a "Do Not Serve" list for bad actors).

The AI had to do five things for each customer:

1. Verify the Job: Are they really working at the university they claim?
2. Check the Company: Is that company a real science lab?
3. Check the Email: Does their email address match their company?
4. Sanctions Check: Are they on a government "Do Not Serve" list?
5. Background Check: Have they done research related to this DNA before?

The Results: The Robot Wins on Speed and Cost

The results were surprising and very promising:

• Accuracy: The best AI (Gemini 2.5 Pro) was just as good as the human experts at spotting the bad guys. It got the "flag" right about 90% of the time, which is basically the same as the humans.
• Speed: The AI did the work in 3 minutes (including a human double-check), while the humans took 15 minutes. The AI was 5 times faster.
• Cost: This is the big one.
  • Human Cost: ~$14.00 per customer.
  • AI Cost: ~$1.18 per customer (including the human double-check).
  • Pure AI Cost: If the AI just gathers the info without a human looking at it first, it costs only $0.23. That's 50 times cheaper than the human!

The Secret Sauce: Tools vs. Just Searching

The researchers tested the AI in two ways:

1. Just Web Search: The AI just Googled things.
2. Web Search + Special Tools: The AI was given special "keys" to unlock government databases, scientific journals, and researcher profiles (like ORCID).

The Lesson: Giving the AI the special keys made it much better at checking sanctions lists (because those lists are hard to find on the open web). However, for finding news articles about industry researchers, the AI was actually better when it just used the open web, because the special tools were too focused on academic papers.

The Catch: The Human Must Still Hold the Keys

The paper emphasizes that while the AI is great at gathering information, it shouldn't be the one to make the final decision.

• The Analogy: Think of the AI as a brilliant research assistant who brings you a stack of files and says, "Here is everything I found about this person."
• The Human Role: The human manager looks at the files and says, "Okay, I see the evidence. I'm going to approve this order" or "No, this looks suspicious, reject it."

If we let the AI decide on its own, it might make mistakes or be biased. But if the AI does the boring, expensive legwork and the human makes the final call, we get the best of both worlds.

Why This Matters

• Biosecurity: It makes it much harder for bad actors to get dangerous DNA because the screening is cheaper and faster, so more companies can afford to do it.
• Science: Legitimate scientists won't be delayed by slow, expensive checks.
• The Future: As the market for synthetic biology grows, we can't rely on human detectives alone. We need AI assistants to handle the heavy lifting, keeping the "bakery" safe without slowing down the good customers.

In short: We found a way to use AI to do the boring, expensive background checks for DNA orders. It's fast, cheap, and almost as accurate as a human expert, but it still needs a human boss to make the final "yes" or "no" decision.

Technical Summary

1. Problem Statement

Context: Synthetic nucleic acids are unique laboratory reagents that can be used to create dangerous pathogens ("Sequences of Concern" or SOCs) without requiring a live pathogen culture. While legitimate research often requires access to these sequences, malicious actors could misuse them.
The Challenge: To prevent misuse, providers must perform legitimacy screening (verifying customer identity and purpose). However, widespread adoption of this screening is limited due to high costs.

• Cost Barrier: Legitimacy screening costs approximately £73 per flagged order, compared to £0.53 for automated sequence screening.
• Process Bottleneck: The "follow-up" phase of screening involves time-consuming, mechanical information gathering (checking institutional affiliations, email domains, sanctions lists, and publication history) which currently relies on manual human labor.
• Regulatory Gap: No jurisdiction explicitly mandates customer screening, leading to inconsistent security practices globally.

Research Question: Can Large Language Models (LLMs) with web search capabilities and specialized tools automate the information-gathering phase of legitimacy screening more efficiently and accurately than human experts?

2. Methodology

Dataset Construction

The researchers created a dataset of 41 customer profiles (extended to 134 for specific metric testing), each paired with a simulated order for an SOC.

• Profile Types:
  • Academic users of SOCs.
  • Industry users of SOCs.
  • Researchers on the U.S. Consolidated Screening List (CSL).
  • General life science researchers (non-SOC background).
• Ground Truth: Established via manual review of publications, patents, and institutional records. All PII was anonymized.
• Task Definition: Screeners were evaluated on five verification tasks:
  1. Institutional affiliation verification.
  2. Institution type verification.
  3. Email domain verification.
  4. Sanctions screening.
  5. Relevant work search (publications/patents).

Experimental Setup

• AI Models Tested: Five LLMs (Claude Sonnet 4, Gemini 2.5 Pro, Grok 4, GLM 4.6, MiniMax M2).
• Conditions:
  1. Web Search Only: Models used a search API (Tavily).
  2. Web Search + Specialized Tools: Models accessed APIs for the U.S. Consolidated Screening List, Europe PMC, and ORCID profiles.
• Human Baseline: Two expert human screeners (co-authors) performed the same tasks. They were given 30 minutes per profile (though snapshots at 5 minutes were also taken to simulate AI speed).
• Evaluation Metrics:
  • Flag Accuracy: Correct identification of "flag," "no flag," or "undetermined."
  • Source Quality: Whether cited sources were independent and editorially overseen (e.g., excluding LinkedIn/Wikipedia).
  • Source Fidelity: Whether claims were directly supported by the cited sources.
  • Work Relevance: For Task 5, whether found work matched the relevance of the reference work.
• Cost Estimation: Calculated based on marginal costs (API tokens, search queries, and human hourly wages of ~$54/hr).

3. Key Contributions

• First Empirical Benchmark: This study provides the first rigorous, head-to-head comparison of LLMs against human experts for biosecurity legitimacy screening.
• Tool Integration Analysis: It evaluates the specific impact of integrating specialized APIs (sanctions lists, academic databases) versus relying solely on general web search.
• Cost-Benefit Framework: It establishes a quantitative framework for the economic viability of AI-assisted screening, breaking down costs by information gathering vs. human review.
• Error Taxonomy: The paper categorizes AI and human errors (e.g., "Criterion Deviation," "Search Failure," "Source Misinterpretation") to identify specific areas for improvement.

4. Results

Performance Accuracy

• Flag Accuracy: The best AI model (Gemini 2.5 Pro with tools) achieved 89.8% accuracy, statistically comparable to the human baseline (89%).
• Source Quality & Fidelity: AI models outperformed humans on these metrics. Humans often cited lower-quality sources (e.g., personal websites) or made unsupported claims, whereas AI models adhered more strictly to source guidelines.
• Task Specifics:
  • Sanctions Screening: Specialized tools (CSL API) significantly improved AI performance over web-only search.
  • Work Relevance: Surprisingly, models with specialized tools (ORCID/Europe PMC) performed worse on finding industry-related work (patents/news) than web-only models, as they relied too heavily on academic databases and missed non-academic sources.

Cost and Efficiency

• Total Cost (AI + Human Review):
  • Human Baseline: $14.04 per customer.
  • Best AI Configuration (Gemini 2.5 Pro + Tools): $1.18 per customer.
  • Reduction: Approximately 12x cheaper (or ~10x on average across models).
• Information Gathering Only:
  • AI models averaged $0.23 per customer.
  • This is roughly 50–60 times cheaper than the human baseline for the data collection phase alone.
• Throughput:
  • Human: ~15.6 minutes per profile.
  • AI (including human review time): ~3.1 minutes per profile.
  • Speedup: ~5x faster.

Error Analysis

• Criterion Deviation (39.7% of errors) was the most common error type for both humans and AI, often due to ambiguous rules (e.g., how to treat researchers from sanctioned countries using personal emails).
• Search Failure was lower in web-only AI models than in tool-enhanced models, suggesting that specialized tools can sometimes narrow the search scope too much, missing relevant non-academic data.

5. Significance and Implications

• Scalability: AI-assisted screening enables DNA synthesis providers to scale verification capacity without proportional increases in labor costs, addressing the bottleneck in current biosecurity practices.
• Human-in-the-Loop (HITL) Model: The paper advocates for a hybrid approach where AI handles the mechanical information gathering and flagging, while humans retain final authority over order fulfillment (ship/reject). This mitigates the risk of AI bias while maximizing efficiency.
• Regulatory Alignment: The proposed HITL model aligns with emerging regulations like the EU Biotech Act and the EU AI Act. Because the AI does not make autonomous decisions affecting individual rights, it avoids classification as a "high-risk" AI system.
• Future Directions: The study suggests that future systems should integrate multiple sanctions databases (beyond the U.S. list) and develop hybrid workflows where AI flags uncertain cases for human review. It also highlights the need to address bias against researchers from low-and-middle-income countries who may have less digital footprint.

Conclusion: The study demonstrates that AI tools, particularly when augmented with specialized APIs, can perform legitimacy screening with accuracy comparable to human experts at a fraction of the cost and time. This supports the immediate piloting of AI-assisted screening in the synthetic biology industry to enhance global biosecurity.