KinConfBench: A Curated Benchmark for Cofolding Models on Kinase Conformational States

The paper introduces KinConfBench, a curated benchmark revealing that while state-of-the-art cofolding models can classify kinase conformations with moderate accuracy, they fail to capture ligand-induced structural diversity and suffer from severe "apo-drift," highlighting the need for new dynamical evaluation metrics beyond geometric pose fitting.

The Gist

Imagine you are trying to design a master key that fits into a very specific, high-tech lock. This lock isn't just a static piece of metal; it's a living, breathing machine that changes its shape depending on what it's holding. Sometimes it's wide open (active), and sometimes it's clamped shut (inactive).

In the world of medicine, these "locks" are proteins called kinases, and the "keys" are drug molecules. If you want to stop a disease, you need a drug that fits perfectly into the lock while the lock is in the right shape.

The Problem: The "Smart" Predictors Are Too Rigid

Recently, scientists have built incredibly powerful AI tools (like Boltz-2, Chai-1, and Protenix) that can predict what these protein locks look like just by reading their genetic code and the shape of the drug key. These tools are like super-smart architects who can draw a blueprint of a building instantly.

However, there's a catch. These AI architects are great at drawing the walls of the building, but they struggle with the doors and windows that need to move. They tend to draw the building in its "default" state (empty and still), even when you tell them a specific piece of furniture (the drug) has been placed inside that should force the room to rearrange itself.

The Solution: KinConfBench (The "Fitness Test")

To see if these AI architects are actually ready for the real world, the authors of this paper created a new, strict exam called KinConfBench.

Think of this as a driving test for self-driving cars.

• The Old Test: Did the car stay in its lane? (This is like checking if the drug fits in the pocket).
• The New Test (KinConfBench): Did the car actually turn the steering wheel correctly when it saw a pedestrian? (This is checking if the protein changed its shape correctly to match the drug).

They gathered 2,225 high-quality examples of human kinase locks and their keys to see how well the AI could predict the shape-shifting behavior.

What They Found: The "Apo-Drift"

The results were a bit of a wake-up call.

1. Good at Geometry, Bad at Dynamics: The AI models were excellent at placing the drug in the right spot (like parking a car in a garage). But, they often failed to realize that the garage door needed to open or the walls needed to shift to accommodate the car.
2. The "Default Mode" Trap: The biggest issue was something the authors call "Apo-Drift." Imagine you ask a robot to build a house with a specific, weirdly shaped sofa inside. Instead of building a room that fits the sofa, the robot just builds a standard empty room and ignores the sofa.
   • Even when the AI knew the drug was there, it kept predicting the protein in its "empty" (ligand-free) state. It was essentially "memorizing" the empty room instead of learning how the room changes when furniture is added.
3. The "All-or-Nothing" Problem: When the AI tried to guess multiple possibilities (like rolling a dice 20 times to see different outcomes), it didn't give a variety of answers. It just gave the same wrong answer over and over again. It lacked the creativity to explore different ways the protein could move.

Why This Matters

In drug discovery, getting the shape right isn't just about aesthetics; it's about safety and effectiveness.

• If a drug is designed for a "closed" lock, but the AI predicts an "open" lock, the drug might fail in the real world.
• The paper argues that we can't just trust the AI because it gets a high score on "fitting the key." We need to trust it to understand the dance between the drug and the protein.

The Takeaway

The authors are saying: "Stop just checking if the puzzle pieces fit together. Check if the picture makes sense when the pieces move."

They are calling for the next generation of AI to stop being static photo-realists and start becoming dynamic animators. Until these models learn to predict how proteins wiggle, twist, and change when a drug arrives, we can't fully rely on them to design the life-saving medicines of the future.

In short: The AI is a brilliant architect, but it needs to learn that buildings aren't just static structures—they are living machines that change shape when you walk inside.

Technical Summary

1. Problem Statement

While data-driven "cofolding" models (e.g., AlphaFold3, Chai-1, Boltz-2) have revolutionized protein structure prediction, their application to Structure-Based Drug Discovery (SBDD) faces a critical gap. Current evaluation metrics primarily focus on:

• Ligand-centric geometry: Root Mean Square Deviation (RMSD) of the ligand pose.
• Global fold confidence: Overall structural similarity to the native state.

These metrics treat the protein receptor as a static scaffold. However, kinases are dynamic molecular switches that toggle between active and inactive conformational states (e.g., DFG-in vs. DFG-out) to regulate function. A ligand often induces a specific conformational change (induced-fit) required for binding. The paper argues that current models may successfully place a ligand in a pocket (low RMSD) while predicting the wrong global conformational state of the kinase, rendering the prediction useless for drug selectivity and lead optimization. There is a lack of benchmarks specifically designed to evaluate a model's ability to capture ligand-induced conformational diversity.

2. Methodology

A. Dataset Curation: KinConfBench

The authors constructed a high-quality benchmark dataset named KinConfBench derived from the RCSB PDB:

• Source: 437 catalytically active human kinase domains.
• Filtering: Applied strict criteria including resolution (< 4.5 Å), exclusion of NMR structures, removal of pseudokinases, and exclusion of phosphorylated chains.
• Final Set: 2,225 structurally unique kinase chains representing 1,420 unique systems (43 apo targets and 1,377 holo complexes).
• Labeling: Used the KinCoRe software suite to assign categorical structural labels based on the work of Modi and Dunbrack. These labels capture:
  • DFG motif orientation (spatial).
  • X-DFG motif torsion angles.
  • $\alpha$C-helix orientation.
  • Salt bridge orientation.
  • Activation loop orientations.
  • Regulatory spine orientation.
  • Activity state: Binary classification (Active vs. Inactive).

B. Models Evaluated

Three state-of-the-art open-source cofolding models were tested:

1. Boltz-2
2. Chai-1
3. Protenix

C. Evaluation Protocol

• Inference: Each model generated an ensemble of N=20 structures per target using standard inference pipelines (MSA generation via ColabFold, default pairing strategies).
• Geometric Filtering: Predictions were first filtered for basic structural validity (Global lDDT-Cα ≥ 0.7, Pocket lDDT-PLI ≥ 0.8, Ligand RMSD < 2.0 Å).
• Conformational Assessment: The primary metric was the KinCoRe label match. A prediction was considered "correct" only if all eight KinCoRe labels matched the experimental ground truth.
• Generalization Test: A "Time-Split" benchmark was performed where holo complexes appearing in the PDB after each model's training cutoff date were treated as test cases to assess generalization to unseen ligand-induced states.

3. Key Results

A. Geometric Success $\neq$ Conformational Correctness

• While all three models achieved high accuracy in geometric metrics (ligand placement and global fold), there was a weak correlation between geometric success and correct conformational state prediction.
• Models frequently produced structures with excellent ligand RMSD but incorrect DFG or $\alpha$C-helix orientations (e.g., predicting an active state when the ligand requires an inactive state).

B. Mode Collapse and Lack of Diversity

• Ensemble Behavior: The models exhibited severe mode collapse. When generating 20 samples, the results were often "all-or-nothing"—either all 20 samples were correct, or all 20 were incorrect.
• Sampling Diversity: Even among "correct" predictions, the structural diversity (standard deviation in distances and dihedral angles) was negligible (~0.1 Å). This indicates the models are not exploring the conformational landscape but are collapsing into a single, narrow energy basin, failing to capture the thermal fluctuations necessary for induced-fit motions.

C. The "Apo-Drift" Phenomenon

• In the time-split generalization test, all three models displayed a strong tendency toward "Apo-Drift."
• When presented with a novel ligand (holo state) not seen during training, the models predominantly predicted the ligand-free (apo) state rather than the ligand-induced state.
• This suggests the models are memorizing the apo-state baselines from the training data rather than learning the physical principles of how ligands modulate protein conformation.

D. Specific Failure Cases

• The paper highlights MAP4K1 (PDB: 7M0M) as a representative failure case where all models failed to induce the correct conformational shift despite perfect ligand placement, settling instead into an alternate DFG-in basin.

4. Key Contributions

1. KinConfBench: The first curated benchmark specifically designed to evaluate cofolding models on kinase conformational states rather than just ligand pose. It includes 2,225 high-quality chains with rigorous KinCoRe labeling.
2. New Evaluation Metrics: Shifts the focus from ligand RMSD to functional conformational fidelity (matching specific DFG, $\alpha$C-helix, and activation loop states).
3. Identification of Generalization Gaps: Demonstrates that current SOTA models suffer from mode collapse and apo-drift, failing to generalize to unseen ligand-induced conformational changes.
4. Critical Analysis of SBDD: Provides evidence that high-confidence geometric scores are insufficient for drug discovery, as they mask critical errors in the protein's regulatory state.

5. Significance and Future Directions

• Impact on Drug Discovery: The findings imply that relying solely on current cofolding models for lead optimization is risky. A model might suggest a drug fits the pocket but fails to stabilize the necessary inactive/active state, leading to false positives in virtual screening.
• Future Requirements: Next-generation models must be trained to:
  • Capture conformational diversity and induced-fit motions.
  • Distinguish between apo and holo states based on ligand physics rather than memorization.
  • Handle post-translational modifications (PTMs), covalent inhibitors, and competitive multi-ligand scenarios.
• Conclusion: KinConfBench establishes a new standard for evaluating whether AI models can truly act as computational assays for rational drug design, emphasizing that conformational correctness is as critical as geometric fit.