💬 NLP

SupraBench: A Benchmark for Supramolecular Chemistry

This paper introduces SupraBench, the first benchmark designed to evaluate large language models on fundamental supramolecular chemistry tasks such as binding affinity prediction and host-guest reasoning, alongside the release of a specialized 16M-token corpus (SupraPMC) to support domain adaptation.

Original authors: Tianyi Ma, Yijun Ma, Zehong Wang, Weixiang Sun, Ziming Li, Connor R. Schmidt, Chuxu Zhang, Matthew J. Webber, Yanfang Ye

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyi Ma, Yijun Ma, Zehong Wang, Weixiang Sun, Ziming Li, Connor R. Schmidt, Chuxu Zhang, Matthew J. Webber, Yanfang Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a custom lock and key system, but instead of metal, you are using tiny, invisible molecules. This is the world of Supramolecular Chemistry. It's like a high-stakes game of "lock and key" where a "Host" molecule tries to grab onto a specific "Guest" molecule to do something useful, like delivering medicine or cleaning up toxins.

Right now, figuring out which lock fits which key is incredibly slow and expensive. Scientists have to run massive computer simulations that can take days for just one pair, or even longer to build them in a real lab.

Enter AI (Large Language Models or LLMs). Scientists hoped these AI chatbots could act as super-fast assistants, predicting which locks fit which keys instantly. But until now, no one had a standardized "driver's test" to see if these AI drivers were actually good at chemistry or just guessing.

That's where this paper comes in. The authors created SUPRABENCH, the first official "driver's test" for AI in this specific field.

The Test Drive: SUPRABENCH

Think of SUPRABENCH as a driving exam with five specific challenges to see how well an AI handles the chemistry road:

The "How Strong?" Test (Binding Affinity): If you give the AI a picture of a lock and a key, can it guess exactly how tightly they will hold hands? (This is the most important test for drug design).
The "Best Match" Test (Top-Binder Selection): If you show the AI one lock and four similar keys, can it pick the best one that fits?
The "Environment" Test (Solvent Identification): Chemical reactions happen in different "liquids" (like water, alcohol, or oil). Can the AI look at the lock and key and guess which liquid they are swimming in?
The "Explain It" Test (Host-Guest Description): Can the AI explain why a lock and key fit together in plain English?
The "Draw It" Test (Molecular Identification): Can the AI look at a 2D drawing of a molecule and correctly type out its chemical name code?

To help the AI study for this test, the authors also released a massive library of 16 million words of chemistry articles (called SUPRAPMC), essentially a "textbook" the AI can read to learn the rules.

The Results: The AI is a Novice Driver

The researchers tested eight different AI models (some free, some expensive) on this exam. Here is what they found:

The "Big Kids" Win, But Still Stumble: The most advanced, expensive AI models (like the latest versions from Google and OpenAI) did the best. However, they still got a lot of questions wrong. There is still a huge gap between what AI can do and what a human expert can do.
The "Study Hacks" Don't Always Work:
- Few-Shot (Showing examples): Giving the AI a few examples of how to answer helped it explain things better, but it actually made the AI worse at predicting how strong the bond would be. It's like showing a student a sample math problem, which helps them write an essay, but confuses them when they try to solve a new equation.
- CoT (Chain of Thought): Asking the AI to "show its work" or explain its reasoning step-by-step seemed like a good idea. But in chemistry, it backfired. The AI started making up confident-sounding facts that were completely wrong. It was like a student confidently writing down the wrong formula because they were trying to sound smart.
Reading the Textbook Helps (But Has a Catch): When they trained the AI on their new chemistry textbook (SUPRAPMC), it got much better at predicting bond strengths (the math part). However, it got worse at following strict formatting rules, like picking the right letter (A, B, C, or D) for multiple-choice questions. It learned the science but forgot the rules of the test.
The "Drawing" Problem: When asked to look at a drawing and write the chemical code, the AI could usually get the general shape right (the "scaffold"), but it often messed up the tiny details (the specific connections between atoms). It's like recognizing a car but drawing the wheels on the roof.

The Bottom Line

The paper concludes that while AI is getting better at chemistry, it is not yet ready to replace human experts or run the lab on its own. The "reasoning" gap is real: AI can sound very confident, but it often lacks the deep, specific knowledge required to get the numbers right.

The authors hope that by releasing this test (SUPRABENCH) and the textbook (SUPRAPMC), other researchers will use them to build better, more reliable chemistry AIs in the future. They are essentially saying, "Here is the ruler we will use to measure progress, and here is the study guide to help you get there."

Technical Summary: SUPRABENCH

Problem Statement

Supramolecular chemistry, focusing on reversible non-covalent host–guest assemblies, is critical for applications ranging from drug delivery to in vivo detoxification. However, the design of these systems remains a bottleneck, requiring extensive "dry-lab" verification (e.g., DFT or MD simulations) that can take days per candidate pair before wet-lab synthesis. While Large Language Models (LLMs) have shown promise in molecular binding tasks, no existing benchmark systematically evaluates their capability in host–guest reasoning across fundamental supramolecular chemistry tasks. Current benchmarks largely focus on single-molecule properties or reaction prediction, leaving the complex interplay of host–guest interactions unaddressed.

Methodology

1. Dataset Construction (SUPRABENCH)

The authors constructed SUPRABENCH, the first benchmark specifically for supramolecular chemistry reasoning.

Data Source: Anchor records were extracted from SupraBank, a public repository of experimentally reported supramolecular interactions.
Preprocessing: A six-step cleaning pipeline was applied to address noise, including numeric parsing, organic-solvent filtering, van't Hoff temperature correction, and outlier removal. This resulted in 4,635 high-quality records covering 2,008 unique components.
Task Definition: The benchmark defines four fundamental reasoning tasks and one auxiliary vision-based task:
1. Binding Affinity Prediction (BAP): A regression task predicting $\log K_a$ given host and guest SMILES strings.
2. Top-Binder Selection (TBS): A 4-way multiple-choice question (MCQ) requiring the model to identify the strongest binder among chemically similar guests.
3. Solvent Identification (SID): A 6-way classification task to infer the solvent environment (e.g., water, DMSO) from molecular structures, as binding constants are solvent-dependent.
4. Host–Guest Description (HGD): An open-ended QA task where models describe physicochemical properties of hosts or guests.
5. Molecular Identification (MI): An auxiliary image-to-SMILES generation task testing the ability to recover molecular identity from 2D structural drawings.

2. Domain Adaptation Corpus (SUPRAPMC)

To support domain adaptation, the authors released SUPRAPMC, a curated 16M-token corpus of supramolecular chemistry articles.

Source: Extracted from Europe PMC (1900–2026).
Filtering: Initial retrieval of 420,950 articles was refined using a rule-based filter combining positive supramolecular keywords with negative biomedical contamination filters (e.g., excluding "host cells"), resulting in 133,867 high-precision articles.

3. Evaluation Protocol

Models: Eight LLMs were evaluated, including open-weight models (Qwen3.5, Llama-3.1, DeepSeek-v4) and proprietary models (GPT-5.4, Gemini-3-Flash).
Prompting Strategies: Three strategies were tested: Base (zero-shot), Few-Shot (3–5 in-domain examples), and Chain-of-Thought (CoT).
Metrics: Task-specific metrics included MAE/RMSE for regression, accuracy/regret for selection, F1/Balanced Accuracy for classification, and ROUGE for generation. For MI, metrics included SMILES validity, exact match, and Tanimoto similarity.

Key Results

1. Performance Gaps and Model Hierarchy

Substantial Headroom: Even the best-performing models leave significant room for improvement. For example, the best Binding Affinity Prediction (BAP) MAE was 1.25, and Top-Binder Selection accuracy plateaued at 0.513.
Frontier Dominance: Gemini-3-Flash consistently achieved state-of-the-art (SOTA) performance across BAP, TBS, and SID tasks. DeepSeek-v4 and GPT-5.4-Mini followed closely.
Task Difficulty: Performance varied sharply by task family. BAP and TBS were generally more challenging than SID for smaller models.

2. Impact of Prompting Strategies

No Universal Strategy: Prompting strategies were highly task-dependent.
- Few-Shot improved performance on Host–Guest Description (HGD) but degraded Binding Affinity Prediction (e.g., Llama3.1-8B MAE increased from 2.699 to 5.504).
- Chain-of-Thought (CoT) consistently hurt performance on BAP and Molecular Identification.
Failure Mode Analysis: In BAP, CoT often led to "confident hallucinations." Models would articulate plausible chemical reasoning (e.g., citing "typical literature values") that resulted in predictions off by orders of magnitude (e.g., predicting $\log K_a \approx 11$ instead of $2$). This suggests CoT amplifies reasoning gaps when domain knowledge is lacking.

3. Domain Adaptation (DAPT)

Regression vs. Format: Fine-tuning on SUPRAPMC improved BAP regression performance (MAE decreased) for both Qwen3.5-9B and Llama3.1-8B.
Trade-offs: However, DAPT degraded performance on Top-Binder Selection and caused the Llama variant to fail at following strict letter-format constraints in Solvent Identification. This indicates that free-form scientific text adaptation transfers well to regression but conflicts with strict MCQ output formats.

4. Molecular Identification

Scaffold vs. Connectivity: Models could recover the general molecular scaffold (high Tanimoto similarity) but frequently failed to capture exact bond-level connectivity (low Canonical SMILES match).
CoT Detriment: Explicit reasoning cues (CoT) destabilized image-to-SMILES generation, reducing exact match rates.

Significance and Claims

The paper positions SUPRABENCH and SUPRAPMC as foundational resources to catalyze research in chemistry-grounded LLMs. The authors claim:

Benchmarking Necessity: SUPRABENCH fills a critical gap by providing a unified protocol to evaluate LLMs on supramolecular host–guest reasoning, a domain previously unaddressed by existing benchmarks.
Current Limitations: Despite SOTA performance by frontier models, supramolecular chemistry remains a genuine bottleneck. Current LLMs lack the rigorous, calibrated reasoning required for accurate affinity prediction and precise structural identification.
Methodological Insights: The study reveals that "reasoning" via CoT is not a substitute for domain knowledge and can be detrimental when the model lacks the underlying chemical facts. Furthermore, domain adaptation requires careful balancing between text exposure and instruction-format preservation.
Future Direction: The results suggest that closing the gap in supramolecular reasoning will likely require targeted domain adaptation and the integration of external chemistry tools rather than relying solely on prompting strategies or general pretraining.

The authors emphasize that these benchmarks are for research evaluation and hypothesis generation, not for driving clinical or regulatory decisions without expert oversight and experimental validation.