How to make the most of your masked language model for protein engineering

This paper introduces a flexible stochastic beam search sampling method for masked language models that optimizes protein properties by evaluating entire-sequence neighborhoods, demonstrating through extensive in silico and in vitro antibody engineering experiments that the choice of sampling strategy is at least as critical as the model itself.

The Gist

Imagine you are a master chef trying to create the perfect new recipe for a dish that cures a specific disease. You have a "Master Cookbook" (a massive AI trained on millions of existing recipes) and a "Base Recipe" (a starting antibody that works okay but needs tweaking).

Your goal is to make small changes to the Base Recipe to make it stronger, safer, and more effective. However, there are two big problems:

1. The Recipe Book is huge: There are billions of possible ways to change the ingredients.
2. Tasting is expensive: You can't just cook every single variation and taste it. You have to send a tiny batch to a real lab to test, which takes weeks and costs a fortune.

This paper is about how to use a computer AI to pick the best variations to send to the lab, so you don't waste time on bad ideas.

Here is the breakdown of their discovery, explained simply:

1. The Old Way: "The Blind Guessing Game"

Previously, scientists used a method called Gibbs Sampling. Imagine you are editing a sentence one word at a time.

• You change word #1, see if it sounds good.
• Then you change word #2, see if it sounds good.
• Then word #3...

The problem is that this is like trying to find the best route through a maze by only looking at the next step. You might get stuck in a dead end, or you might miss a shortcut because you were too focused on the immediate next word. Also, doing this one word at a time is slow and computationally heavy.

2. The New Way: "The Whole-Page Scan" (Stochastic Beam Search)

The authors propose a smarter way: Stochastic Beam Search.

Instead of changing one word at a time and waiting, imagine you take the whole page of the recipe and ask the AI: "If I swap out these 5 ingredients at once, how good does the entire dish look?"

• The Magic Trick: The AI is incredibly fast at calculating how "good" a full sentence (or protein sequence) is, even if it's just a tiny change away from the original.
• The Beam: Instead of looking at just one path, the AI looks at a "beam" of 5 or 20 different variations simultaneously. It keeps the best ones and discards the bad ones, then expands those winners into new variations.
• The "Stochastic" Part: To make sure they don't all end up with the exact same boring recipe, they add a little bit of "random noise" (like shaking the spice jar). This ensures they get a diverse set of candidates, not just clones of each other.

Analogy: Think of it like a search party looking for a hidden treasure.

• Old Way: One person walks forward one step, checks the ground, turns, walks another step. Very slow.
• New Way: A helicopter flies over a wide area, spots 20 promising spots, and sends a team to check all of them at once.

3. The "Taste Test" (Guidance)

Sometimes, you don't just want a tasty dish; you want it to be healthy (low calories) and cheap to make. In the paper, these are "scoring functions" (like checking if the antibody is stable or if it might cause an allergic reaction).

The new method is flexible. It can say: "Okay, keep the AI's favorite recipes, but if a recipe is too 'spicy' (risky), down-rank it. If it's 'cheap' (easy to make), boost it."

They found that using this "taste test" guidance was just as important as the AI model itself. Even a slightly weaker AI model, if guided well, can beat a super-powerful AI model that is just guessing blindly.

4. The Big Surprise

The researchers tested this on real antibodies in a real lab (not just on a computer).

• Result: The method they invented (Stochastic Beam Search) produced significantly more successful antibodies than the old methods.
• The "ESM-2" Surprise: They used a model trained on all proteins (like a general chef who knows everything about food) and it worked surprisingly well for antibodies (a specific type of dish), even though they had models trained only on antibodies.

The Takeaway for Everyone

If you are trying to design a new drug or protein:

1. Don't just tweak one thing at a time. Look at the whole picture.
2. Use a "Beam" approach. Explore many options at once, not just one path.
3. Add a "Guide." Don't just trust the AI's gut feeling; tell it what specific goals you have (safety, cost, stability) and let it balance those goals.

In short: They turned the process of drug discovery from "shooting in the dark one bullet at a time" into "firing a guided missile swarm that hits the target every time."

Technical Summary

Here is a detailed technical summary of the paper "How to Make the Most of Your Masked Language Model for Protein Engineering," published at the GEM workshop, ICLR 2026.

1. Problem Statement

The field of protein engineering, particularly for antibody therapeutics, faces a combinatorial explosion of possible sequence mutations. While numerous protein language models (PLMs), specifically Masked Language Models (MLMs), have been released, there is a significant gap in understanding how to best sample from these models to optimize biological properties.

Current challenges include:

• Inefficient Sampling: Existing methods for MLMs often rely on "mutation-centric" iterative procedures (e.g., denoising or Gibbs sampling) which are computationally expensive ($O(EL^3)$) and tend to produce low-quality, dysfunctional sequences.
• Guidance Limitations: Incorporating additional scoring functions (e.g., binding affinity, immunogenicity risk, isoelectric point) is difficult. Many existing methods require scoring functions to accept partially masked sequences or rely on differentiable gradients, which excludes many practical, non-differentiable metrics.
• Lack of Benchmarking: There is a scarcity of systematic, head-to-head evaluations of sampling algorithms, both in silico and in vitro.

2. Methodology

The authors propose a sequence-centric approach that reframes generation as a search problem over the entire sequence space, leveraging the efficiency of MLMs in evaluating pseudo-perplexity.

A. Stochastic Beam Search with Temperature Annealing

Instead of generating mutations token-by-token, the method evaluates full sequences using the MLM's Pseudo-Log-Likelihood (PLL).

• The "Free" Neighbor Insight: Computing the exact PLL for a single sequence requires $L$ forward passes ($O(L^3)$). However, once the PLL for a template sequence is computed, the approximate PLLs for all 1-edit neighbors (sequences with one substitution) can be derived with negligible additional cost.
• Approximation: The method uses the Wild-Type Marginal Approximation. For a neighbor sequence $x'$ differing from template $x$ at position $k$, the PLL is approximated by using the exact conditional probability at $k$ and the template's conditional probabilities for all other positions. This reduces the cost of evaluating neighbors to $O(L^3)$ amortized, offering a massive speedup over mutation-centric methods.
• Stochastic Beam Search (SBS): To balance likelihood and diversity, the authors employ SBS (Kool et al., 2019). They add Gumbel noise to the scores before ranking. A temperature parameter ($\tau$) scales the likelihood term, allowing control over the trade-off between high-probability sequences and diversity.

B. Multi-Objective Optimization (MOO) with Gradient-Free Guidance

The framework treats the MLM and any external scoring functions as black boxes.

• Black Box Compatibility: Since the method evaluates clean sequences, it can incorporate non-differentiable scores (e.g., OASis percentile for immunogenicity) and scores that require full sequences.
• Scalarization: The authors utilize Smooth Tchebycheff Scalarization (STS) to combine multiple objectives (e.g., PLL, binding probability, synthesizability). Unlike Pareto Non-Dominated Sorting (NDS), STS allows for weighted objectives, aiming to improve performance across all objectives simultaneously rather than finding a Pareto frontier.

3. Key Contributions

1. Novel Sampling Algorithm: Introduction of a flexible, computationally efficient Stochastic Beam Search for MLMs that exploits the structure of the 1-edit neighborhood.
2. Comprehensive Benchmarking: The first extensive head-to-head evaluation of sampling methods and models on real antibody therapeutics campaigns, covering both in silico predictions and in vitro wet-lab validation.
3. Guidance Framework: A robust method for integrating non-differentiable, multi-objective scoring functions without requiring gradient access or partial-sequence inputs.
4. Empirical Insights: Demonstration that the choice of sampling algorithm is at least as impactful as the choice of the language model itself.

4. Experimental Results

In Silico Evaluation

• Models Tested: Nine MLMs (including ESM-2, Sapiens, AbLang-2, AMPLIFY) and three Causal Language Models (CLMs).
• Findings:
  • Beam Search vs. Gibbs: The proposed beam search consistently outperformed Gibbs sampling and denoising methods in predicted synthesizability and diversity metrics.
  • Model Performance: AbLang-2 (trained on antibody data) and ESM-2 650M (trained on generic proteins) showed the best performance, suggesting generic models can be highly effective for antibody optimization.

In Vitro Evaluation (Wet Lab)

• Setup: Tested on a fresh FAb antibody therapeutic campaign. 289 samples across 13 methods were synthesized and screened for synthesizability and binding.
• Key Outcomes:
  • Success Rates: AbLang-2 combined with Stochastic Beam Search achieved the highest success rates.
  • Impact of Guidance: Using supervised models to rank or guide the beam search significantly improved outcomes.
    • STS Guidance: Achieved a 100% success rate for synthesizability and binding when using Smooth Tchebycheff Scalarization with supervised guidance.
  • Comparison: Beam search outperformed Gibbs sampling across all models where both were tested. Gibbs-argmax struggled to propose unique sequences.
  • Developability: Guidance methods produced antibodies with tighter binding affinities and less variance in yield compared to unguided methods.
  • Surprising Finding: ESM-2 (generic training) produced highly "human" sequences, while AbLang-2 (antibody-trained) sometimes produced less human sequences, highlighting that training data alone does not guarantee specific developability traits without proper sampling guidance.

5. Significance and Recommendations

The paper fundamentally shifts the paradigm for protein engineering from mutation-centric generation to sequence-centric search.

Practical Recommendations:

1. Use Supervision: Whenever labeled data is available, use it for ranking and guidance (e.g., STS or NDS) to drastically improve success rates.
2. Model Selection: AbLang-2 is recommended for antibody engineering, though ESM-2 650M is a strong, effective alternative even without antibody-specific training.
3. Sampling Strategy: Adopt Stochastic Beam Search over Gibbs or denoising sampling. It is faster, produces higher-quality sequences, and handles constraints better.
4. Multi-Objective Handling: Prefer Smooth Tchebycheff Scalarization (STS) over Pareto sorting when the goal is to simultaneously satisfy multiple objectives (e.g., binding + stability + low immunogenicity).
5. Caution with Guidance: Supervised guidance can introduce biases (e.g., reducing diversity or altering germline bias); these must be monitored and potentially counter-balanced with specific metrics (e.g., humanness scores).

Conclusion:
This work demonstrates that optimizing the sampling strategy is as critical as the model architecture itself. By enabling efficient, sequence-centric search with flexible guidance, the proposed method significantly accelerates the discovery of viable antibody therapeutics, bridging the gap between theoretical language model capabilities and practical biological success.