💬 NLP

Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation

This paper introduces Speak-to-Structure (S^2-Bench), the first benchmark designed to evaluate Large Language Models on open-domain, one-to-many natural language-driven molecule generation tasks, alongside OpenMolIns, a dataset that enables fine-tuned models to outperform leading LLMs in realistic molecular design.

Original authors: Jiatong Li, Junxian Li, Weida Wang, Yunqing Liu, Changmeng Zheng, Yatao Bian, Dongzhan Zhou, Xiao-yong Wei, Qing Li

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiatong Li, Junxian Li, Weida Wang, Yunqing Liu, Changmeng Zheng, Yatao Bian, Dongzhan Zhou, Xiao-yong Wei, Qing Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: From "Guessing the Answer" to "Designing the Solution"

Imagine you are a chef. For years, researchers have tested AI chefs by giving them a specific recipe (like "Make a chocolate cake") and asking them to recreate a specific cake that already exists in a database. If the AI makes a cake that looks exactly like the one in the database, it gets a gold star.

The Problem: Real cooking (and real science) isn't about copying. If you ask a chef, "Make me a cake that is low in sugar but still tastes sweet," there isn't just one correct answer. There are hundreds of different recipes that could work. The old tests were too strict; they only rewarded the AI for memorizing the "right" answer, not for being creative.

The Solution: This paper introduces S2-Bench (Speak-to-Structure). It's a new test designed to see if AI can act like a real creative chef. Instead of asking for a copy, it asks the AI to invent new things based on a description, knowing that there are many possible "correct" answers.

The Three Challenges (The "Menu")

The paper tests the AI on three specific tasks, which are like different levels of a cooking challenge:

Molecule Editing (The "Tweak" Challenge):
- The Analogy: Imagine you have a specific car model. The test asks the AI to "Add a turbocharger" or "Remove the sunroof."
- The Goal: The AI must take the original car and make only that specific change without breaking the engine or turning the car into a boat. It tests if the AI understands the rules of how things fit together.
Molecule Optimization (The "Upgrade" Challenge):
- The Analogy: You have a car, and you want it to be "faster" or "more fuel-efficient."
- The Goal: The AI needs to modify the car to improve a specific trait (like speed) while keeping the car recognizable. It's not enough to just build a new, fast car from scratch; it has to be an improvement on the original one.
Customized Molecule Generation (The "Blank Canvas" Challenge):
- The Analogy: You tell the AI, "Build me a vehicle with exactly 4 wheels, a red body, and a V8 engine."
- The Goal: The AI has to invent a completely new vehicle from nothing that fits these strict rules. This is the hardest test because the AI has to follow the rules perfectly without any starting template.

The New Training Manual: OpenMolIns

To help the AI get better at these tasks, the authors created a massive new training book called OpenMolIns.

Old Way: Previous training books had very few examples, and they were mostly "Question: What is this molecule? Answer: [Exact Molecule Name]." This taught the AI to memorize.
New Way: OpenMolIns is like a massive library of 1.2 million examples where the AI learns the logic of chemistry. It teaches the AI: "If you want to make something faster, try adding this part," rather than just "Here is the answer."

The Result: Using this new training book, a relatively small AI model (Llama3.1-8B) was able to beat much larger, more famous models (like GPT-4o and Claude-3.5) on these creative tasks. It proved that good training is more important than just having a huge brain.

How They Grade the AI (The Scorecard)

In the old tests, if the AI wrote a molecule that matched the target word-for-word, it got 100%. But in this new test, that's not enough. The authors created a smarter scoring system:

Did you follow instructions? (Success Rate)
Is it a reasonable change? (Similarity)
- The Catch: If you asked to "add a wheel" to a car, and the AI built a completely different car that just happened to have a wheel, the old test would say "Good job!" The new test says, "No, you didn't actually modify the original car; you just ignored the prompt and built something else."
Is it new? (Novelty)
- For the "Blank Canvas" challenge, the AI gets points for inventing something that doesn't already exist in the world's databases.

Key Takeaways

Memory isn't enough: Current AI models are good at recalling facts but struggle to follow complex, creative instructions in chemistry.
One-to-Many is real: In science, one question often has many valid answers. Old tests didn't allow for this; the new one does.
Small models can win: With the right training data (OpenMolIns), a smaller, cheaper AI can outperform massive, expensive ones at designing molecules.
The "Human" Check: The authors had human experts look at the AI's work. They found that the new scoring system (which rewards logical changes over random guesses) matched human opinions much better than the old "exact match" system.

In short, this paper builds a better gym for AI to practice chemistry. Instead of just memorizing flashcards, the AI is now learning how to actually design and build new things based on a conversation.

Technical Summary: Speak-to-Structure (S2-Bench)

Problem Statement

Current Large Language Models (LLMs) show promise in natural language-driven molecule discovery, yet existing evaluation benchmarks suffer from a critical limitation: they rely on one-to-one mappings between textual descriptions and single, pre-defined target molecules (e.g., ChEBI-20, PubChem324K). This paradigm assumes a unique solution for every prompt, which misaligns with the reality of molecular discovery where multiple distinct molecules can satisfy the same biological or physical properties. Consequently, current benchmarks primarily test a model's ability to memorize and recall specific patterns rather than its capacity for creative, open-ended molecular design and genuine structural reasoning. There is a lack of standardized evaluation for "one-to-many" relationships in text-to-molecule generation.

Methodology

1. Speak-to-Structure (S2-Bench)

The authors propose S2-Bench, the first benchmark designed to evaluate LLMs in open-domain natural language-driven molecule generation. Unlike previous datasets, S2-Bench adopts a one-to-many paradigm, where a single instruction can be fulfilled by multiple valid molecular structures. The benchmark consists of three primary tasks, each targeting a different stage of the drug discovery process:

Molecule Editing (MolEdit): Tests the ability to perform precise, localized structural modifications (adding, deleting, or substituting functional groups) while preserving the core scaffold. This mimics lead optimization.
Molecule Optimization (MolOpt): Evaluates goal-oriented chemical reasoning where the model must modify a lead compound to optimize specific properties (LogP, MR, QED) without altering the core structure unnecessarily.
Customized Molecule Generation (MolCustom): Challenges the model to synthesize a novel molecule from scratch based on natural language constraints regarding atom counts, bond types, or functional groups, mimicking the initial design phase.

Evaluation Metrics:
To address the "one-to-many" nature, the authors move beyond exact string matching. They employ an automated evaluation system using cheminformatics toolkits (RDKit) to calculate:

Validity: Whether the generated SMILES string is chemically valid.
Success Rate (SR): Whether the molecule meets the specific constraints (e.g., correct property direction or functional group).
Similarity (for MolEdit/MolOpt): Tanimoto similarity using Morgan Fingerprints to ensure the output is a rational modification of the input, not a random valid molecule.
Novelty (for MolCustom): Measures how distinct the generated molecule is from a reference database (Zinc-250K).
Weighted Success Rate (WSR): A composite metric combining Success Rate with Similarity (for editing/optimization) or Novelty (for customization) to provide a holistic performance score.

2. OpenMolIns Dataset

To facilitate training for these open-ended tasks, the authors introduce OpenMolIns, a large-scale instruction-tuning dataset constructed programmatically from the PubChem database.

Scale: Contains up to 1.2 million instruction-molecule pairs.
Construction: Uses automated chemical toolkits to generate diverse prompts and corresponding valid molecules, ensuring a one-to-many relationship.
Structure: Mirrors the three S2-Bench tasks and includes five data scales (light to xlarge) to study data scaling laws.
Integrity: Designed with zero overlap with the S2-Bench test set (Zinc-250K) to prevent data leakage.

3. Experimental Setup

The authors benchmarked 31 LLMs, categorized into:

Proprietary models (e.g., GPT-4o, Claude-3.5).
Open-source general LLMs (e.g., Llama3, Mistral, Qwen).
Domain-specific LLMs (e.g., ChemLLM, BioT5).
Models fine-tuned on OpenMolIns.

Evaluation was conducted using automated pipelines, with a subset of results validated by human experts to correlate automated metrics with human judgment.

Key Results

Performance Gap in Structural Constraints: Even top-tier proprietary models (e.g., Claude-3.5, Gemini-1.5-Pro) achieved moderate average weighted success rates (~35%), but struggled significantly with MolCustom tasks, failing to surpass a 25% weighted success rate on individual subtasks. This indicates a difficulty in satisfying fine-grained structural constraints (exact atom/bond counts) despite general reasoning capabilities.
Superiority of Instruction Tuning: While pre-trained chemical LLMs (e.g., BioT5-base) performed poorly (4.21% WSR) compared to general LLMs, instruction tuning on OpenMolIns yielded dramatic improvements.
- Llama3.1-8B-Instruct fine-tuned on the largest OpenMolIns scale (xlarge) achieved a 39.33% WSR, outperforming all other models, including larger proprietary ones.
- Remarkably, the small Galactica-125M model, when fine-tuned on OpenMolIns, achieved a 25.73% WSR, outperforming models two orders of magnitude larger (e.g., Llama3-70B).
Limitations of One-to-One Benchmarks: Models fine-tuned on traditional one-to-one datasets (ChEBI-20) performed worse than general LLMs on S2-Bench. These models tended to generate chemically valid but structurally unrelated molecules to satisfy constraints, revealing a reliance on pattern recall rather than structural reasoning.
Data Scaling Laws: The relationship between data volume and performance is task-dependent.
- MolCustom (de novo synthesis) showed significant gains (286%) when scaling from "large" to "xlarge" data.
- MolEdit and MolOpt showed diminishing or negative returns with increased data for smaller models, suggesting these tasks are more constrained by model capacity than data volume.
Validation of WSR: Human expert evaluation confirmed that the Weighted Success Rate (WSR) correlates significantly better with expert judgment (Spearman $\rho \approx 0.90$ ) than raw Success Rate (SR), validating the necessity of penalizing structurally unrelated outputs.

Significance and Claims

The paper claims to establish a new paradigm for evaluating LLMs in molecular discovery by shifting the focus from pattern recall to realistic molecular design.

Bridging the Gap: S2-Bench addresses the mismatch between current evaluation metrics and the open-ended nature of real-world molecule discovery.
Demonstrating Potential: The results demonstrate that with appropriate instruction tuning on large-scale, diverse data, smaller LLMs can achieve state-of-the-art performance in open-domain molecule generation, challenging the notion that only massive models or specialized architectures are viable.
Diagnostic Tool: The benchmark serves as a diagnostic tool to reveal systematic weaknesses in current LLMs regarding discrete, globally coupled structural constraints, guiding future research in reinforcement learning and instruction tuning.

The authors emphasize that their work provides a robust foundation for developing more capable, explainable, and trustworthy LLMs for natural language-driven molecule discovery, moving the field beyond simple text-to-structure translation.