Technical Summary: EpiBench

Problem Statement

The discovery of therapeutic antibodies relies fundamentally on understanding epitopes—the specific regions on an antigen where an antibody binds. This binding determines downstream therapeutic properties, including functional blockade, immune escape, and therapeutic specificity. While specialized deep learning models have been developed to predict epitopes from antigen sequences or structures, they are typically optimized for narrow, single-task prediction targets. Conversely, general-purpose Large Language Models (LLMs) have demonstrated strong biomedical reasoning capabilities but their ability to infer epitope information directly from antigen and antibody sequences to support the broader antibody drug development workflow remains unverified. Existing protein benchmarks focus on general biological properties or fitness landscapes but do not evaluate epitope-centered decisions such as binding site localization, antibody grouping, functional interpretation, and escape assessment within a drug discovery context.

Methodology

To address this gap, the authors introduce EpiBench, a closed-book, sequence-based, and automatically scorable benchmark designed to evaluate epitope reasoning in LLMs.

Benchmark Design Principles

EpiBench is constructed to simulate the practical workflow of antibody development, adhering to five key principles:

1. Workflow Alignment: Tasks correspond to key stages from candidate epitope discovery to escape assessment.
2. Evidence Grounding: Labels are derived from experimentally supported sources, including structural antibody–antigen complexes, curated functional B-cell assays, and deep mutational scanning (DMS) measurements.
3. Closed-Book Formulation: All identifiers (PDB IDs, antibody/antigen names, accession numbers, metadata) are removed to prevent direct lookup or memorization of specific records. Only sequence-level evidence is retained.
4. Task-Adapted Formats: Outputs are formatted as residue predictions, multiple-choice questions, or binary judgments to match specific biological decision points.
5. Shortcut Control: Sampling strategies control answer distributions, candidate orders, and distractor selection to minimize evaluation artifacts based on sequence similarity or mutation identity.

Dataset Composition

EpiBench contains 1,609 curated samples derived from four sources: AsEP, SAbDab, IEDB, and DMS escape measurements. The dataset covers five connected tasks:

• Task 1: Targetable Region Discovery: Predicting a limited set of candidate epitope residues from an antigen sequence without a specific antibody (Antigen-only).
• Task 2: Antibody-Conditioned Epitope Identification: Identifying which candidate epitope region is bound by a specific antibody sequence (Antigen + Antibody + Candidates).
• Task 3: Epitope Binning: Determining if two antibodies bind overlapping antigen regions (Antigen + Two Antibodies).
• Task 4: Functional Epitope Assessment: Distinguishing epitopes associated with functional outcomes (e.g., neutralization, inhibition) from those with only binding evidence (Antigen + Candidates + Function).
• Task 5: Antibody Escape Assessment: Predicting whether a specific antigen point mutation disrupts recognition by a particular antibody (Antibody + Wild-type Antigen + Mutation).

Evaluation Protocol

The authors evaluated nine general-purpose LLMs (including reasoning and non-reasoning models from various providers) under a unified zero-shot protocol. The evaluation included:

• Comparison against random baselines and specialized epitope prediction models (where applicable).
• Stratification by antigen sequence length to test long-context reasoning.
• Comparison between explicit reasoning (Chain-of-Thought) and implicit (direct answer) settings.
• Failure mode analysis of incorrect reasoning traces.

Key Results

Overall Performance

Current LLMs capture partial epitope-related signals but remain limited in precise interface discrimination and antibody-specific grounding.

• Task 1 (Region Discovery): Models showed improved ability to prioritize broadly relevant antigen regions (RegionRecall@50) but performed near random on AUROC, indicating difficulty in distinguishing epitope from non-epitope residues across the full sequence.
• Task 2 & 5 (Conditioned Identification & Escape): Performance was limited, suggesting that recognizing generic antigen properties is easier than grounding specific antibody–antigen interactions or escape effects from sequence alone.
• Task 3 (Binning): Showed the largest gain over random baselines, likely because models could leverage pairwise sequence similarity cues without requiring explicit localization of the binding interface.
• Task 4 (Functional Assessment): Models struggled to connect epitopes to downstream functional evidence, reflecting the difficulty of inferring latent structural contacts and functional consequences not directly expressed in sequences.

Sensitivity and Reasoning Analysis

• Antigen Length: Performance on Task 1 degraded significantly as antigen length increased (from <200 to >800 residues), indicating that LLMs struggle to preserve fine-grained residue position information in long contexts. Tasks 2–4 showed more stable performance across lengths but did not demonstrate robust epitope understanding.
• Explicit Reasoning: The benefit of Chain-of-Thought (CoT) prompting was inconsistent. While it helped Task 1 for some models, it did not reliably improve performance on Tasks 2–5, which require complex sequence-grounded decisions.
• Failure Modes: Analysis of incorrect traces revealed that models often rely on indirect biological proxies rather than consistent epitope relationships. Common failure patterns included:
  • Motif Fixation: Over-reliance on recognizable sequence motifs or functional sites even when insufficient.
  • Knowledge Claims: Appealing to memorized associations rather than analyzing the provided sequences.
  • Similarity Shortcuts: Using CDR or germline similarity as a proxy for epitope binning.

Significance and Claims

The paper positions EpiBench not merely as a leaderboard but as a diagnostic testbed for measuring and improving sequence-aware biomedical LLMs.

• Current Limitations: The results clarify that current LLMs are not yet capable of reliably supporting the full epitope-centered antibody discovery workflow. They lack robustness in long-context residue localization, antibody-specific sequence grounding, and biologically grounded reasoning for functional or escape outcomes.
• Diagnostic Value: By evaluating epitope reasoning across tasks aligned with drug discovery, EpiBench identifies specific bottlenecks (e.g., long-range sequence grounding, residue-level correspondence) that future biological sequence reasoning models must address.
• Standardization: The benchmark provides a standardized, automatically scorable framework that bridges the gap between specialized epitope prediction and general protein language modeling, offering a practical environment for developing reliable LLM-assisted antibody discovery tools.

The authors conclude that while LLMs show promise in capturing coarse signals, significant advancements are required before they can be trusted for critical decisions in antibody drug design.