MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation

This paper introduces MolLangBench, a comprehensive benchmark for evaluating language-prompted molecular structure recognition, editing, and generation, which reveals that even state-of-the-art models like GPT-5 struggle significantly with these fundamental chemical tasks despite their intuitive simplicity for humans.

The Gist

Imagine you have a very smart, well-read robot assistant. You can ask it to write a poem, solve a math problem, or summarize a news article, and it does a great job. But what happens if you ask this robot to act like a chemist?

Specifically, what if you say: "Look at this molecule. Now, swap this specific part for a different one to make it stronger," or "Draw a brand new molecule that looks like a key for a specific lock"?

This paper, MolLangBench, is essentially a "report card" for AI on how well it can do these chemistry tasks. The researchers found that while our AI is getting smarter, it's still struggling to understand the precise language of molecules.

Here is a breakdown of the paper using simple analogies:

1. The Three Big Tests

The researchers created a benchmark (a standardized test) with three main challenges, mirroring what a real chemist does in a lab:

• The "Spot the Difference" Test (Recognition):

  • The Task: You show the AI a picture of a molecule (or a string of code representing it) and ask, "How many carbon atoms are exactly three steps away from this nitrogen?"
  • The Analogy: Imagine showing a robot a complex Lego castle and asking, "Count the red bricks that are exactly three bricks away from the blue tower."
  • The Result: Even the smartest AI (GPT-5) got this right only about 86% of the time. For humans, this is easy; for AI, it's surprisingly hard.

• The "Edit the Recipe" Test (Editing):

  • The Task: You give the AI a molecule and an instruction: "Take off the methyl group and add a hydroxyl group here."
  • The Analogy: Imagine a chef who knows a recipe perfectly. You tell them, "Swap the salt for sugar." A good chef does it. The AI often swaps the salt, but accidentally adds sugar to the wrong bowl, or forgets to remove the salt entirely.
  • The Result: The AI got this right about 85% of the time. It's okay, but in chemistry, a tiny mistake can make a drug toxic instead of helpful.

• The "Invent a New Dish" Test (Generation):

  • The Task: You describe a molecule in words ("A six-sided ring with a nitrogen at the top and a double bond on the right...") and ask the AI to draw or write the code for it.
  • The Analogy: You describe a specific car to a mechanic: "It needs four wheels, a V8 engine, and a red paint job." The mechanic hands you a drawing of a bicycle with a red engine.
  • The Result: This was the hardest. The best AI only got this right 43% of the time. Most of the time, it created "impossible" molecules that couldn't exist in real life.

2. Why is the AI failing?

The paper identifies a few funny but frustrating reasons why the AI stumbles:

• The "Token" Problem (Counting is Hard):
  Modern AI reads text like we read words, not like we count individual letters. If you ask a human, "How many 'r's are in 'strawberry'?", they might pause. AI struggles even more with this. In chemistry, atoms are like letters. The AI often merges two atoms into one "word" (token) in its mind, causing it to lose track of where things are. It's like trying to count the bricks in a wall, but the wall is painted so the bricks look like one giant block.

• The "Visual Illusion" Problem:
  The researchers tried giving the AI pictures of molecules instead of code. They thought, "Maybe if it sees the molecule, it will understand better!"

  • The Analogy: It's like showing a robot a photo of a car and asking it to build a working engine. The robot draws a picture that looks like a car, but the wheels are floating, and the engine is inside the trunk. It looks right from a distance, but up close, the physics are broken.

3. The "Gold Standard" Dataset

To make sure the test was fair, the researchers didn't just ask the AI to guess. They hired real chemistry experts to create the questions and answers.

• They treated the dataset like a gold standard.
• Every question was checked by multiple experts to ensure there was only one correct answer.
• They made sure the instructions were so clear that a human with basic chemistry knowledge could follow them without guessing.

4. The Big Takeaway

The main message of the paper is: We are overestimating how good AI is at chemistry right now.

While AI is amazing at writing stories or coding software, it is currently a "novice" at chemistry. It can recognize patterns, but it lacks the precision to manipulate molecular structures safely.

• Why does this matter? If we want AI to help discover new life-saving drugs or clean energy materials, it needs to be 100% accurate. A 43% success rate in creating molecules means we can't trust it to build new medicines yet.

Summary

Think of MolLangBench as a driving test for AI.

• The Test: Can you recognize the car, change a tire, and build a new engine from a description?
• The Score: The AI can recognize the car (86%), it can change a tire mostly correctly (85%), but if you ask it to build a new engine from a description, it often builds a toaster instead (43%).

The paper calls for more research to teach AI the "grammar" of molecules so it can stop making "impossible" chemical structures and start helping scientists solve real-world problems.

Technical Summary

1. Problem Statement

Current Artificial Intelligence (AI) systems face a critical gap in bridging natural language instructions with precise molecular structure manipulation. While Large Language Models (LLMs) excel at reasoning and natural language understanding, and specialized molecular models (based on SMILES or graphs) perform well on domain-specific tasks (e.g., property prediction), there is a lack of robust systems that can:

1. Recognize detailed structural features of a molecule from text or image prompts.
2. Edit a molecule based on specific, unambiguous natural language instructions (e.g., "replace the hydroxyl group with a methoxy group at the para position").
3. Generate a valid molecular structure from scratch based on a detailed textual description.

Existing benchmarks often suffer from ambiguity (where one description maps to multiple structures) or lack the precision required for chemical tasks, where minor errors in stereochemistry or atom connectivity render a molecule invalid or biologically inactive. Furthermore, current models struggle to translate high-level chemical concepts into exact structural representations.

2. Methodology

Dataset Construction (MolLangBench)

The authors constructed a comprehensive benchmark consisting of three core tasks, ensuring high-quality, deterministic, and unambiguous outputs through rigorous curation:

• Molecular Structure Recognition:

  • Source: 18 diverse tasks derived from the UniChem database (178M molecules).
  • Automation: Ground truth labels were generated using automated cheminformatics tools (RDKit).
  • Tasks: Includes local topology (n-hop neighbors, bond types), functional group detection (amides, furans, etc.), and stereochemistry (R/S, E/Z).
  • Format: Models must output both categorical/numerical answers and specific atom indices to ensure structural localization.
  • Scale: 200 instances per task (totaling 3,600+ samples).

• Language-Prompted Molecule Editing:

  • Source: Molecules from UniChem (<40 non-hydrogen atoms).
  • Annotation Pipeline: A rigorous multi-step process involving expert chemists:
    1. Writing: Annotators create unambiguous modification instructions (e.g., ring fusion, group substitution).
    2. Peer Review: A second annotator verifies clarity and uniqueness.
    3. Independent Validation: Two validators reconstruct the molecule from the instruction using chemical drawing software. The result must match the ground truth exactly.
  • Scale: 200 core validated instances + 200 extended instances.

• Molecule Generation from Structural Descriptions:

  • Process: Similar to editing, but annotators write detailed structural descriptions (skeleton, substituents, stereochemistry) that must allow a chemist to reconstruct the molecule uniquely.
  • Validation: Validators reconstruct the molecule solely from the text description to ensure a one-to-one mapping.
  • Scale: 200 core validated instances + 200 extended instances.

Modality Support: The benchmark evaluates models across three input modalities:

1. Text: SMILES strings.
2. Image: Molecular structure images (with atom indices labeled).
3. Graph: (Preliminary evaluation included, though limited by current model availability).

Evaluation Metrics

• Recognition: Accuracy of the answer and Localization Accuracy (correct identification of specific atom indices).
• Editing/Generation:
  • SMILES Validity: Is the output a chemically valid string?
  • Structural Accuracy: Does the generated molecule match the ground truth (Tanimoto similarity and exact match)?
  • Pass@k: Evaluation of success rates over multiple attempts.

3. Key Contributions

1. MolLangBench Benchmark: The first comprehensive benchmark specifically designed to evaluate the molecule-language interface with a focus on precision, unambiguity, and deterministic outputs.
2. Rigorous Annotation Protocol: Introduction of a multi-stage expert validation pipeline that eliminates the ambiguity common in previous text-molecule datasets, ensuring that every prompt has exactly one correct structural answer.
3. Multi-Modal Evaluation: Support for evaluating models on SMILES, molecular images, and (conceptually) molecular graphs, allowing for a holistic view of cross-modal capabilities.
4. Comprehensive Baseline Analysis: Extensive evaluation of state-of-the-art (SOTA) models, including GPT-4o, GPT-5, o-series reasoning models, DeepSeek-R1, and vision-language models (GPT Image 1, o4-mini).

4. Key Results

The evaluation reveals significant limitations in current AI systems, even the most advanced ones:

• Recognition Tasks:

  • GPT-5 (the strongest model) achieved 86.2% average accuracy.
  • Localization Gap: There is a notable drop between recognizing a feature (e.g., "count benzene rings") and localizing the specific atoms (e.g., "list indices of benzene atoms"). GPT-5 dropped from 92.5% recognition to 84.0% localization.
  • Stereochemistry: Models struggle significantly with stereochemical reasoning (E/Z, R/S), with many models performing worse than random guessing on these tasks.
  • Tokenization Issue: The authors attribute localization errors to Byte-Pair Encoding (BPE) tokenization, where multiple atoms are grouped into a single token, hindering precise atom-level counting.

• Editing Tasks:

  • GPT-5 achieved 85.5% accuracy.
  • Gemini-2.5-Pro and GPT-5 led the field, but performance dropped significantly for other models.
  • Error Types: Common failures included invalid SMILES syntax, stereochemical mismatches, and incorrect chain lengths or ring fusions.

• Generation Tasks:

  • Performance is drastically lower. GPT-5 achieved only 43.0% accuracy.
  • Validity: Only ~69% of GPT-5's generated SMILES strings were chemically valid.
  • Vision-Language Models: The image-generation model GPT Image 1 performed poorly, achieving only 13.5% accuracy in editing and 0% in generation. While images looked visually plausible, they often violated fundamental chemical rules (e.g., wrong valency, incomplete rings).

• Downstream Impact:

  • Experiments showed that prompting models to first describe the molecular structure before predicting properties (e.g., BBBP, BACE) improved downstream prediction accuracy by >5%, suggesting that structural understanding is a prerequisite for chemical reasoning.

5. Significance and Future Directions

• Highlighting the "Chemical Reasoning" Gap: The paper demonstrates that while LLMs have strong general reasoning, they lack the precise, atom-level reasoning required for chemistry. Current models are "hallucinating" structures rather than manipulating them with chemical rigor.
• Need for Specialized Data: The results suggest that general-purpose pretraining is insufficient. Future models require training data that explicitly aligns natural language with precise molecular representations (SMILES, graphs, images) to bridge the modality gap.
• Tokenization Limitations: The paper identifies BPE tokenization as a fundamental bottleneck for atom-level tasks, suggesting a need for atom-wise tokenization or specialized architectures for chemical domains.
• Benchmark Standard: MolLangBench sets a new gold standard for evaluating chemical AI, moving beyond simple property prediction to complex, instruction-following structural manipulation.

In conclusion, MolLangBench exposes that current AI systems are not yet reliable for autonomous chemical design or modification. Significant advancements in model architecture, training data, and tokenization strategies are required before AI can effectively replace or augment human chemists in structural design workflows.