Benchmarking Generative Large Language Models for de novo Antibody Design and Agentic Evaluation

This study demonstrates that five compact, open-source LLM-inspired transformer architectures, when trained from scratch on 15 million antibody sequences and fine-tuned on disease-specific repertoires, achieve uniformly high generative fidelity and structural quality for de novo antibody design, indicating that at this parameter scale, performance is driven primarily by data and scale rather than specific architectural family differences.

The Gist

Imagine the immune system as a massive, high-tech security force. Its soldiers are antibodies, tiny Y-shaped proteins that hunt down viruses and bacteria. Designing a new antibody from scratch is like trying to invent a new key that fits a specific lock (a virus) without ever seeing the lock before. Usually, scientists have to guess and test millions of keys, which is slow and expensive.

This paper is about teaching AI to be a master locksmith who can instantly design the perfect key. But instead of just using one AI, the researchers wanted to know: Does it matter which "brain" architecture we use, or is it all about the data we feed it?

Here is the story of their experiment, broken down into simple parts:

1. The Five Different "Brains"

The researchers took five different types of modern AI models (inspired by famous families like Llama, Gemma, and Mistral). Think of these as five different types of super-intelligent apprentices.

• They didn't just download these apprentices; they trained them from scratch.
• They fed them a massive library of 15 million existing antibody "keys" (from a database called Observed Antibody Space).
• The Goal: See if the type of apprentice matters, or if they all just become experts because they read the same library.

2. The Result: All Apprentices Became Masters

After training, the researchers tested the apprentices. They asked them to invent brand new antibodies.

• The Surprise: It didn't matter which "brain" architecture they used. All five apprentices performed almost exactly the same. They were equally creative, equally unique, and equally good at making new designs.
• The Lesson: At this specific size (compact models), the quality of the library (the training data) and the size of the brain mattered way more than the specific blueprint of the brain itself. It's like saying that if you give five different chefs the same 15 million recipes, they will all cook equally delicious meals, regardless of whether they are French or Italian trained.

3. The Real-World Test: Fighting Viruses

Next, they put the apprentices to work on real problems. They asked them to design antibodies for four dangerous enemies: SARS-CoV-2 (Coronavirus), HIV, HER2 (a cancer marker), and Ebola.

• The Structural Check: They used super-computers to fold these digital antibodies into 3D shapes. The results were stunning: the shapes were incredibly stable and perfect (scoring nearly 93 out of 100 on a stability scale).
• The Docking Test: They simulated how well these new antibodies would "hug" the viruses. The results showed they would bind very tightly, effectively neutralizing the threat.

4. Safety and Novelty Checks

Before a new drug can be used, it must be safe and unique.

• Is it new? Yes. The AI-designed antibodies had loops (the part that grabs the virus) that were completely different from anything seen in nature before.
• Is it safe? Yes. The antibodies looked very "human," meaning our bodies wouldn't reject them as foreign invaders. They also checked to ensure the antibodies wouldn't accidentally trigger an allergic reaction or attack healthy cells.

5. The "AI Manager" (Agentic Evaluation)

Finally, the researchers introduced a new tool: an AI Manager.

• Imagine the five apprentices are the workers, and this new AI (powered by a model called Claude) is the foreman.
• This foreman automatically checks the workers' designs, runs the safety tests, and picks the best candidates to present to human scientists. This speeds up the process from months to minutes.

The Big Takeaway

This paper tells us that for designing new medicines, we don't need to obsess over finding the "perfect" AI architecture. Instead, we should focus on feeding our AI the best possible data. Even smaller, compact AI models can be incredibly powerful if they are trained correctly.

It's like realizing that you don't need a Ferrari engine to win a race; if you have the best map (data) and a skilled driver (training), even a reliable sedan can get you to the finish line faster than anyone else. This opens the door for more labs to design life-saving antibodies without needing super-expensive, massive supercomputers.

Technical Summary

1. Problem Statement

Despite significant progress in computational antibody engineering, there is a critical gap in the systematic evaluation of modern open-source Large Language Model (LLM) backbone families specifically for de novo single-domain antibody (sdAb/VH) design. Key unanswered questions include:

• Do architectural differences between prominent LLM families (e.g., Llama, Gemma, Mistral) significantly impact generative performance at compact model scales?
• Is the "family-specific" design of these models a determining factor in antibody generation, or is performance driven primarily by training data and parameter count?
• How can these models be integrated into automated, rigorous evaluation pipelines for biological validation?

2. Methodology

The study employed a rigorous, multi-stage experimental design involving five distinct transformer architectures customized for antibody generation:

• Model Selection & Architecture: Five compact transformer variants were selected, inspired by prominent open-source LLM families: Llama4, Gemma3, DeepSeekV3, Mistral 7B, and NVIDIA Nemotron3.
• Pretraining Strategy:
  • All five models were trained from scratch (not fine-tuned from existing weights) to eliminate pre-training bias.
  • Dataset: 15 million sequences from the Observed Antibody Space (OAS) database.
• Fine-Tuning:
  • Models were subsequently fine-tuned on disease-stratified repertoires to target specific pathogens:
    • SARS-CoV-2: 4,688 sequences
    • HIV: 430 sequences
    • HER2: 22,778 sequences
    • Ebola virus: 2,868 sequences
• Structural & Binding Validation:
  • Structure Prediction: Top-ranked candidates were evaluated using four state-of-the-art structure prediction tools: AlphaFold-2, Boltz-2, RoseTTAFold-2, and ESMFold.
  • Docking: Computational docking was performed to estimate binding free energies.
• Biological Rigor Checks:
  • Novelty: IMGT-defined CDR-H3 extraction and BLASTp assessment against database hits.
  • Humanness: Analysis of VH germline content.
  • Immunogenicity: Profiling via NetMHCIIpan 4.3 to detect MHC-II binding risks.
• Agentic Evaluation: A proof-of-concept pipeline was developed using the ModelContext Protocol (MCP) and Claude Sonnet 4.6 to automate structural profiling and candidate prioritization.

3. Key Results

A. Generative Fidelity & Diversity

Pretraining yielded uniformly high performance across all five architectures, indicating that architectural differences were negligible at this scale:

• Sequence Diversity: 0.507 – 0.516 (Coefficient of Variation [CV] = 0.8%).
• Uniqueness: Approached 1.0 (near-perfect uniqueness).
• Novelty: 0.925 – 0.977 (CV = 2.2%).

B. Structural Quality

Structural assessment of top candidates revealed high-confidence folds with no statistically significant differences between the model families:

• Mean pLDDT Scores: Ranged from 92.88 (SD 1.54) to 93.77 (SD 2.16).
• Statistical Analysis: Kruskal-Wallis test showed $H=2.06$, $p > 0.05$ (N=100), confirming that the choice of LLM family did not significantly alter structural prediction quality in this single-seed experiment.

C. Binding & Biological Safety

• Binding Affinity: Predicted binding free energies ranged from -36.34 to -65.60 kcal/mol (noting a likely data formatting anomaly in the source text regarding "-8722", the valid range suggests strong binding potential).
• Novelty: CDR-H3 identity to closest database hits was 0% – 29%, confirming high novelty.
• Humanness: VH germline content was 77% – 90%, indicating strong consistency with human germline sequences.
• Immunogenicity: No strong MHC-II binders were detected across CDR regions, suggesting the generated antibodies are immunogenically silent.

4. Key Contributions

1. First Systematic Benchmark: Provides the first comprehensive comparison of modern open-source LLM backbones for de novo sdAb design, establishing a baseline for future research.
2. Architectural Insight: Demonstrates that at compact model scales, generative capacity is driven by training data and model scale rather than specific family architectural elements. This suggests that for antibody design, resource allocation should prioritize data curation and scale over complex architectural tuning.
3. Agentic Workflow: Introduces a novel MCP-based agentic pipeline (leveraging Claude Sonnet 4.6) that automates the transition from sequence generation to structural profiling and prioritization, streamlining the design cycle.
4. High-Quality Candidates: Successfully generated novel, human-consistent, and structurally stable antibody candidates for four distinct disease targets with validated binding potential.

5. Significance

This study fundamentally shifts the paradigm for computational antibody engineering by proving that compact, open-source LLMs can achieve state-of-the-art performance without requiring massive, proprietary architectures. The finding that architectural nuances are less critical than data and scale at this parameter regime allows researchers to focus on data quality and dataset diversity. Furthermore, the integration of an agentic evaluation pipeline represents a significant step toward fully automated, AI-driven drug discovery workflows, reducing the manual burden of validating generative models for therapeutic applications.