Benchmarking open-source tools for in silico antiviral drug discovery

This paper advocates for increased investment in antiviral drug discovery by presenting a comprehensive survey of open-source tools, a curated dataset of 43,005 viral protein-ligand interactions, and a benchmark of 15 AI-based models that demonstrates the superior performance of fine-tuned machine learning approaches like Boltz-2 and DrugFormDTA for predicting binding affinities.

The Gist

The Big Picture: Racing Against a Virus

Imagine a new virus shows up at the door. The authors of this paper argue that while vaccines are great, they take time to build and don't work for everyone. Antiviral drugs are the "fire extinguishers" we need right now. They can be deployed quickly, especially if we can find existing drugs that already work against other viruses and just use them for this new one (a process called drug repurposing).

However, there's a problem: We don't have a good map of which drugs work against which viruses. This paper is an attempt to build that map using computers.

The Problem: A Messy Library

To teach a computer how to predict if a drug will kill a virus, you need a massive library of data: "Drug X fits into Virus Protein Y."
The authors went to the biggest library available, called BindingDB, to get this data. But they found the library was a disaster.

• The "Polyprotein" Puzzle: Many viruses (like SARS-CoV-2) write their instructions as one giant, long string of text (a polyprotein) that needs to be cut into smaller, functional pieces. The library had thousands of entries where the data was attached to the whole giant string instead of the specific cut piece (the actual target).
• The Fix: The authors acted like librarians cleaning up a mess. They manually (and with AI help) cut those giant strings into the correct pieces. They found that 31% of the viral data was unusable until they did this "cutting." Once cleaned, they had a high-quality dataset of 43,005 drug-protein interactions.

The Test: A Race Between Tools

Once they had their clean data, they wanted to see which computer tools were the best at predicting if a drug would stick to a virus. They set up a race with 15 different open-source tools (free software anyone can use).

Think of these tools as different types of detectives trying to solve a puzzle:

1. The Docking Detectives: These tools try to physically simulate how a drug molecule fits into a virus protein, like trying to fit a key into a lock. They use physics and geometry.
   • The Winner: GNINA was the best at this. It's like a detective with a very good 3D model of the lock.
2. The AI Predictors: These tools use machine learning (AI) to look at patterns. They don't necessarily build a 3D model; they just look at the "shape" of the data and guess.
   • The Winners: Boltz-2 and DrugFormDTA were the best here.
   • The Surprise: The authors took their own cleaned data and used it to "train" (teach) the DrugFormDTA model. This was like giving the detective a specific study guide for this virus. The result? The model got much smarter, jumping from a correlation score of 0.5 (a coin flip) to 0.7 (a strong prediction).

The Results: No Single "Magic Bullet"

The paper tested these tools on 853 different drugs across 10 different viruses.

• The Takeaway: There is no single tool that wins every time.
  • Boltz-2 was great at predicting how drugs bind to HIV, but it struggled with SARS-CoV-2 (likely because the "polyprotein" mess mentioned earlier confused it).
  • GNINA (the docking tool) was very consistent but slower.
  • DrugFormDTA (the AI tool) became the champion after being trained on the authors' specific, cleaned-up data.

The Toolkit They Built

Beyond just testing tools, the authors built a few resources for other scientists to use:

1. A Clean Dataset: A curated list of 43,000+ viral drug interactions, fixed and ready for use.
2. A Drug Library: A list of approved drugs, safe natural compounds, and investigational antivirals.
3. A Dashboard: A website (antivirals-database.radvac.org) where people can look up these drugs.

What They Didn't Say

It is important to stick to what the paper actually claims:

• They did not discover a new cure for a virus.
• They did not test these drugs on humans or animals in this study.
• They did not claim that one specific tool is perfect for the future.
• They simply showed that cleaning the data makes the computers work better, and that different tools have different strengths depending on the specific virus.

Summary Analogy

Imagine you are trying to predict which keys open which locks in a massive, messy warehouse.

1. The Old Way: You grab a pile of keys and locks from the warehouse, but many locks are still taped together in giant bundles. You try to guess which key fits, but you keep failing because the locks are the wrong size.
2. This Paper's Work: The authors went in, cut all the bundles apart, and organized the locks correctly.
3. The Experiment: They gave this organized pile to 15 different "guessing machines" (some use physics, some use AI).
4. The Result: They found that the AI machine learned the fastest when it was taught using their newly organized pile. They also found that the best machine for one type of lock (HIV) wasn't necessarily the best for another (Coronavirus).

The paper concludes that if we want to be ready for the next pandemic, we need to invest in better data cleaning and better computer tools to find these "keys" faster.

Technical Summary

Technical Summary: Benchmarking Open-Source Tools for In Silico Antiviral Drug Discovery

Problem Statement

The paper addresses the critical gap in pandemic preparedness: the lack of FDA-approved antivirals for the majority of viral families with pandemic potential. While vaccines are a primary defense, they face limitations including long development timelines, public hesitancy, and the inability to protect against pathogens that are difficult to vaccinate against (e.g., HIV, RSV). Furthermore, adaptive immunity takes weeks to develop, leaving a window of vulnerability.

The authors argue that drug repurposing and the design of antiviral combinations offer the fastest path to deploying effective treatments during an outbreak. However, this strategy is hindered by:

1. Data Quality Issues: Existing public datasets (e.g., BindingDB) contain significant noise, including unprocessed viral polyproteins that are unsuitable for machine learning (ML) training.
2. Methodological Variance: There is a lack of comprehensive benchmarks comparing open-source computational tools (ML-based, docking, and hybrid) specifically on viral protein targets.
3. Economic Barriers: Pharmaceutical incentives for repurposing are low due to limited market exclusivity and the complexity of licensing combination therapies.

Methodology

1. Data Curation and Polyprotein Processing

The authors constructed a custom, high-quality dataset of 43,005 viral protein-ligand binding measurements by aggregating data from BindingDB, SMACC, and Heli-SMACC.

• Polyprotein Challenge: A critical finding was that 31% of viral binding data in BindingDB required polyprotein sequences to be carefully split before use. Many entries listed targets as large polyproteins (e.g., SARS-CoV-2 ORF1ab) rather than the specific functional domains (e.g., Main Protease) used in assays.
• Resolution: The team developed a mapping function to link polyprotein slices and ligand names to specific protein targets, utilizing LLMs (Anthropic's Haiku) to verify references.
• Dataset Composition: The final dataset includes 43,005 records for training and a hold-out test set of 280 records (Ki/Kd values only) for initial model tuning.

2. Model Fine-Tuning

The authors fine-tuned the DrugFormDTA model (Khokhlov et al., 2025), a sequence-based ML model using Chemformer for molecules and ESM-2 for proteins.

• Training Strategy: They trained four separate models and ensembled them, replicating the original architecture. They tested both MSE and Huber loss functions with different learning rate schedules.
• Goal: To adapt a general DTA model specifically for viral proteins, addressing the bias in original training data which heavily favored mammalian proteins.

3. Benchmarking Framework

The authors evaluated 15 open-source tools on a larger, diverse test set of 853 antiviral compounds spanning 16 protein targets from 10 virus species (including SARS-CoV-2, HIV, HCV, Dengue, Zika, Influenza).

• Tool Categories:
  • ML/Sequence-based: DrugFormDTA, Boltz-2, GatorAffinity, Interformer, DrugCLIP, Uni-Mol+GNINA.
  • Docking/Physics-based: GNINA, FlowDock, DiffDock, Protenix-Dock, AutoDock-GPU, Vina-GPU, Uni-Dock.
• Metrics: Performance was assessed using Pearson correlation ($r$), Spearman rank correlation ($\rho$), $Q^2$, RMSE, MAE, AUROC, and BEDROC.
• Data Handling: Raw scores were linearly recalibrated to match $pK_d$ units to ensure fair comparison across tools with different output scales.

Key Results

1. Impact of Data Cleaning and Fine-Tuning

• DrugFormDTA Performance: Fine-tuning DrugFormDTA on the cleaned, polyprotein-split antiviral dataset significantly improved performance. The Pearson correlation ($r$) increased from 0.50 (base model) to 0.70 (fine-tuned ensemble).
• Polyprotein Necessity: The results underscore that failing to split polyproteins in training data severely degrades model generalizability to specific viral targets.

2. Benchmarking Open-Source Tools

On the 853-compound test set, performance varied significantly by tool type and target:

• Top ML Performers: Boltz-2 and the fine-tuned DrugFormDTA ranked highest among ML-based approaches.
  • Fine-tuned DrugFormDTA achieved $r = 0.701$.
  • Boltz-2 achieved $r = 0.316$ overall, though it performed exceptionally well on HIV Reverse Transcriptase ($r = 0.68$), likely due to abundant training data for that specific protein.
• Top Docking Performers: GNINA (CNN scoring) performed best among docking approaches ($r = 0.302$), outperforming standard Vina and other rigid docking tools.
• General Trends:
  • Variance: No single tool performed best across all viral proteins. Performance was highly target-dependent (e.g., Boltz-2 struggled with SARS-CoV-2 MPro, likely due to the polyprotein issue in its training data).
  • Sequence vs. Structure: Sequence-only models (DrugFormDTA) generally outperformed structure-based docking methods in this specific benchmark, though docking tools like GNINA remained competitive.
  • Failure Rates: Some tools (e.g., DiffDock in the authors' initial tests) failed to place ligands in active sites without specific pocket guidance, resulting in near-zero correlation.

3. Resource Compilation

The authors compiled and released:

• A library of 2,096 approved small molecule drugs and 3,311 total approved drugs (including international approvals).
• A comprehensive dataset of GRAS (Generally Recognized As Safe) compounds and natural products derived from COCONUT, LOTUS, FooDB, and other sources.
• A public dashboard at antivirals-database.radvac.org containing investigational and approved antivirals.

Significance and Claims

The paper positions itself as a foundational resource for the open-source community in antiviral discovery. Its primary claims are:

1. Data Integrity is Paramount: The study demonstrates that the "garbage in, garbage out" principle is particularly acute in viral drug discovery due to polyprotein complexities. The authors claim that careful curation (splitting polyproteins) is a prerequisite for training effective ML models.
2. Open-Source Viability: The work provides empirical evidence that open-source tools, when properly benchmarked and fine-tuned on relevant data, can achieve performance comparable to or exceeding commercial standards for specific viral targets.
3. Foundation for Rapid Response: By providing a cleaned dataset, a benchmark of tools, and a library of repurposable compounds, the authors aim to lower the barrier to entry for researchers developing rapid drug repurposing pipelines and combination therapies.
4. Modest Scope: The authors explicitly state that while their work provides a foundation, it does not claim to have solved the problem of de novo drug design. They emphasize that the field still lacks a single "best" approach and that medicinal chemistry expertise remains crucial for final selection. They note that their benchmarks are specific to the datasets and targets tested and do not guarantee generalizability to all future viral outbreaks without further adaptation.

The paper concludes that investing in these technologies and techniques is essential for future pandemic preparedness, enabling the rapid identification of antivirals and the design of synergistic drug combinations.