🧬 biology

Mechanistic Interpretability of Antibody Language Models Using SAEs

This paper investigates the use of TopK and Ordered Sparse Autoencoders (SAEs) to interpret and steer autoregressive antibody language models, finding that while TopK SAEs are better for mapping biological concepts, Ordered SAEs are more effective for precise generative steering.

Original authors: Rebonto Haque, Oliver M. Turnbull, Anisha Parsan, Nithin Parsan, John J. Yang, Anna L. Beukenhorst, Charlotte M. Deane

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rebonto Haque, Oliver M. Turnbull, Anisha Parsan, Nithin Parsan, John J. Yang, Anna L. Beukenhorst, Charlotte M. Deane

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The "Secret Language" of Antibodies: A Simple Guide

Imagine you are trying to understand a massive, complex library filled with millions of instruction manuals. These manuals aren't written in English, but in a specialized "biological code" that describes how to build antibodies—the tiny soldiers in your immune system that hunt down viruses and bacteria.

Scientists have built "AI Librarians" (called Antibody Language Models) that have read all these manuals. These AI librarians are incredibly good at writing new manuals (generating new antibodies), but there is a huge problem: they are "black boxes."

Even though the AI can write a perfect manual, we don't actually know how it’s thinking. Is it actually understanding the biology, or is it just guessing based on patterns? If we want to use AI to design life-saving drugs, we can't just hope it's right; we need to be able to look under the hood and steer the AI like a car.

This paper explores a new way to "peek inside" the AI's brain using a tool called Sparse Autoencoders (SAEs).

The Two Tools: The "Highlighter" vs. The "Master Architect"

The researchers tested two different ways of using these SAE tools to understand the AI. Think of it like trying to understand a complex piece of music.

1. The TopK SAE (The "Highlighter")

Imagine you give a student a textbook and tell them, "Highlight only the 32 most important words on every page."

What it does: It finds specific, tiny details. It can point to a specific "word" in the antibody code and say, "This part belongs to this specific family of antibodies." It’s very visual and easy to understand—like seeing a bright neon highlight on a specific sentence.
The Flaw: Just because you can highlight a word doesn't mean you can change the story. If you try to "steer" the AI by shouting that highlighted word louder, the AI gets confused. It’s like highlighting the word "Apple" in a cookbook; you can see the word, but shouting "APPLE!" doesn't actually change the recipe from cake to pie. It’s interpretable (you see it), but not steerable (you can't control it).

2. The Ordered SAE (The "Master Architect")

Now, imagine instead of highlighting words, you look at the themes of the book. Instead of "Apple," this tool looks for the concept of "Fruit" or "Dessert."

What it does: It organizes information in a hierarchy. It looks at the big, abstract ideas first (the "Master Plan") and then the tiny details later.
The Benefit: This tool is a steering wheel. If you tell the AI, "Give me more of this 'Fruit' concept," the AI actually changes its output. It starts writing recipes that are more dessert-like. It allows scientists to say, "Make this antibody more like this specific family," and the AI actually does it.
The Trade-off: It’s harder to "see" exactly where the idea is. It’s not a neat neon highlight on a single word; it’s more like a general "vibe" that spreads across the whole page. It’s steerable, but less visually simple.

Why Does This Matter?

In the world of medicine, "close enough" isn't good enough. If we are designing an antibody to fight a new virus, we need to be able to precisely tune its properties—making it more stable, more soluble, or more effective at binding to a target.

The big takeaway from this paper is:

We can see the AI's "thoughts": We proved that the AI actually does learn biological rules (like which "family" an antibody belongs to).
We found the steering wheel: We discovered that while the "Highlighter" method is great for seeing what the AI knows, the "Master Architect" method is what we need if we want to actually command the AI to design specific, useful medicines.

By mastering these "steering wheels," scientists are moving closer to a future where we don't just guess how to fight diseases, but we can precisely program the biological tools to win the war.

Technical Summary: Mechanistic Interpretability of Antibody Language Models Using SAEs

1. Problem Statement

The rapid advancement of antibody language models (ALMs) has revolutionized antibody drug discovery, enabling tasks like library generation, humanization, and light-chain optimization. However, these models operate as "black boxes." This lack of interpretability presents several critical challenges:

Trust & Safety: Difficulty in verifying if models rely on biologically relevant features or spurious correlations.
Overfitting Detection: Inability to discern if a model has truly learned protein physics/biology or merely memorized training data.
Generative Control: The inability to precisely steer model outputs toward specific desired properties (e.g., specific germline gene identities or developability traits) during the design process.

2. Methodology

The researchers investigated the use of Sparse Autoencoders (SAEs) to decompose the dense, polysemantic activations of ALMs into human-interpretable, sparse latent features.

A. Models Evaluated:

Primary Model: p-IgGen (a 17M parameter GPT-like decoder-only model).
Robustness Models: IgLM (13M parameters) and ProGen2-OAS (764M parameters) to ensure findings generalize across different scales and architectures.

B. SAE Architectures:

TopK SAEs: These limit the number of active latents to a fixed $k$ . They are designed to find sparse, localized features but often suffer from "feature splitting" or "absorption," where features are not easily steerable.
Ordered SAEs (O-SAEs): A hierarchical approach using nested grouping and monotonically decreasing truncation weights. This forces the model to learn "abstract" global features at lower indices and "fine-grained" details at higher indices.

C. Evaluation Pipeline:

Feature Identification: Used Linear Probes (logistic regression) to see if SAE latents could predict biological concepts: CDR identity (residue-level) and V/J germline gene identity (sequence-level).
Steering: Implemented "activation patching" by adding scaled decoder vectors back into the model's hidden states ( $h^*_l = h_l + \alpha \cdot d^{(i)}$ ) to observe if the model's generated output shifted toward a target concept.
Validation: Measured the quality of steered sequences using Hamming distance (novelty), cosine distance (diversity), and ESM-2 likelihood (protein-likeness).

3. Key Contributions

First Application to Autoregressive ALMs: The paper marks the first use of SAEs to interrogate autoregressive, domain-specific antibody language models.
The Interpretability-Steerability Trade-off: The study identifies a fundamental tension: TopK SAEs are excellent for mapping and visualizing biological concepts, but Ordered SAEs are superior for controlling them.
Generalizability Proof: Demonstrated that SAE-based feature discovery and steering are robust across models of vastly different sizes (13M to 764M parameters).

4. Results

TopK SAE Performance:
- High Interpretability: TopK latents successfully captured CDR and J-gene identities. Latent activations were visually sparse and localized to specific biological regions (e.g., CDRH3), whereas raw neurons were "polysemantic" and scattered.
- Low Steerability: Despite high correlation with biological concepts, TopK latents failed to reliably steer the generation of specific genes in p-IgGen.
Ordered SAE Performance:
- High Steerability: O-SAEs successfully identified hierarchical features. For example, steering on a specific low-index latent significantly increased or decreased the proportion of the IGHJ4 gene in generated libraries.
- Abstract Nature: O-SAE features were less "localized" to specific residues than TopK features, suggesting they capture higher-level, long-range sequence instructions necessary for generative control.
Generative Quality: Steered libraries produced sequences that were more novel and diverse than natural sequences but remained on the "plausible antibody manifold" (as confirmed by ESM-2 likelihoods).

5. Significance

This work provides a roadmap for Rational Antibody Design. By moving from "black-box" generation to "steerable" generation, researchers could theoretically use SAEs to:

Target Specificity: Force the model to generate antibodies with specific germline frameworks.
Optimize Developability: Steer generation toward sequences with high solubility or stability.
Enhance Trust: Provide a mechanistic explanation for why a model suggests a particular sequence, bridging the gap between deep learning and structural biology.