GermRL: Alleviating The Germline Bias In Autoregressive Antibody Language Models Through Reinforcement Learning

Technical Summary: GermRL

Problem Statement
Antibody therapeutics rely on sequence diversity generated during development to achieve antigen specificity. While large language models (LLMs) trained on antibody repertoires have advanced the generation and screening of novel candidates, they suffer from a significant limitation: a strong "germline bias." These models tend to generate sequences that remain too close to their germline origins, failing to explore the broader sequence space necessary for discovering mutations that encode desirable therapeutic properties. While previous research has addressed this bias in masked antibody language models, the issue remains unaddressed in generative autoregressive language models, which are critical for one-shot antibody design.

Methodology
To address this gap, the authors introduce GermRL, a lightweight and modular reinforcement learning (RL) framework designed to alleviate germline bias in pre-trained autoregressive antibody language models. The core technical approach involves:

• Algorithm: The framework utilizes Group Relative Policy Optimization (GRPO) to fine-tune pre-trained models.
• Objective: The RL agent is optimized to generate antibodies that satisfy specific mutation thresholds relative to their germline sequences while maintaining structural plausibility.
• Novel Modifications: Within the GRPO implementation, the authors introduce a specific pair of modifications tailored to antibody applications. These changes are designed to increase training efficiency and specifically discourage "reward hacking," a common failure mode in RL where agents exploit loopholes in the reward function rather than learning the intended task.

Key Results
GermRL demonstrates a marked improvement over the baseline pre-trained language model in generating diverse antibodies:

• Performance at Mutation Thresholds: Under strict testing conditions, GermRL achieves a pass@1 score of 0.992 for sequences requiring a minimum of 5 mutations from the germline and 0.950 for sequences requiring 35 mutations. In contrast, the unmodified pre-trained model scores only 0.398 and 0.034, respectively, at these same thresholds.
• Sequence Analysis: Comparative analysis between RL-generated sequences and natural antibody sequences reveals that the RL-optimized model successfully explores alternative evolutionary mutational patterns and residue compositional strategies.
• Preservation of Properties: Despite promoting diversity, the generated sequences retain key global properties of natural antibodies, including identifiable germline assignments, embedding-level similarity to natural repertoires, and comparable developability profiles.

Significance and Claims
The paper positions GermRL as a promising framework for navigating the antibody sequence landscape. By effectively promoting diversity from the germline through RL-based optimization, the method enables the exploration of novel candidates that remain biologically plausible. The authors claim that this approach is crucial for advancing AI adoption in therapeutic workflows, as it allows for the discovery of mutations necessary for designing effective antibody therapeutics that existing generative models fail to produce due to inherent biases. The work specifically highlights the ability to generate diverse, high-quality candidates in a single-shot manner, bridging the gap between AI generation and the complex diversity required for antibody discovery.