Technical Summary: MolSight

Problem Statement

While Molecular Large Language Models (LLMs) have advanced drug discovery by learning from textual representations like SMILES, they struggle to fully capture the visual semantics of molecular structures. Existing Molecular Vision-Language Models (VLMs) attempt to bridge this gap but face significant limitations:

1. Structural Misalignment: Standard vision encoders, pre-trained on general images, capture patterns like color and texture rather than the sparse, fine-grained features of atoms and chemical bonds.
2. Topological Loss: Vision tokens preserve spatial locality in the image but fail to encode the adjacency relationships inherent in molecular graphs. Consequently, current VLMs lack the necessary topological modeling to accurately understand molecular structures, leading to poor performance in tasks like image-to-SMILES translation and bioactivity prediction compared to specialized Optical Chemical Structure Recognition (OCSR) tools.

Methodology

To address these issues, the authors propose MolSight, a graph-aware vision-language framework designed to inject chemical-bond adjacency information into visual representations. The model takes a molecular image and its corresponding SVG text (containing explicit structural annotations) as inputs.

The core innovation lies in a Graph Topology Adaptation Layer situated between the vision encoder and the LLM decoder, comprising two complementary modules:

1. Molecular Topology Module (MTM):

   • Function: Injects 2D molecular graph topology directly into vision tokens.
   • Mechanism: It employs a learnable edge predictor to estimate chemical-bond connectivity between vision tokens, generating a soft adjacency matrix. This matrix acts as a topological mask during multi-head graph attention message passing, forcing the model to aggregate information based on predicted molecular adjacency rather than just spatial proximity.
   • Supervision: The edge predictor is supervised by ground-truth bond adjacency matrices derived from the SVG or RDKit parsing.

2. Molecular Grounding Module (MGM):

   • Function: Aligns topology-aware vision tokens with symbolic chemical semantics.
   • Mechanism: It utilizes cross-modal attention where vision tokens (queries) attend to SVG text tokens (keys and values). This allows vision tokens to absorb explicit information regarding atom types, bond connectivity, and positional data provided in the SVG.

Training Strategy:
The model follows a two-stage training approach:

• Stage 1 (Pretraining): The topology adaptation layer is progressively pre-trained. Substage 1 activates the MTM with a combined task loss and edge supervision loss to learn bond connectivity. Substage 2 activates the MGM to align SVG semantics with visual features using only the task loss.
• Stage 2 (Fine-tuning): The model is fine-tuned on downstream tasks using LoRA, with the vision encoder and adaptation layer frozen.

Key Contributions

1. SVG-Guided Structural Alignment: The authors introduce the use of SVG text as a guidance signal to explicitly provide element and connectivity information, addressing the structural misalignment between standard VLM visual representations and molecular semantics.
2. Dual-Module Architecture: The design of the MTM and MGM creates a graph topology adaptation layer that enables general-purpose VLM backbones to capture chemical semantics by learning topology from vision tokens and aligning them with structural annotations.
3. Unified Performance: MolSight is presented as a unified multi-modal model that outperforms existing generalist VLMs, molecular specialist VLMs, and specialized tools across diverse chemical visual understanding tasks.

Experimental Results

The authors evaluated MolSight on four distinct types of chemical visual understanding tasks:

• Image-to-SMILES Translation: MolSight achieved an average Tanimoto similarity of 0.97 and an accuracy of 0.84, significantly outperforming generalist VLMs (e.g., Qwen-VL, LLaVA) and molecular specialist VLMs. Notably, it surpassed specialized OCSR tools like MolScribe and DECIMER in accuracy while maintaining a unified architecture.
• Molecular Captioning (MoleculeQA): On a benchmark testing understanding of structure, source, property, and application, MolSight achieved the highest scores across all categories (e.g., 78.38% on Structure), outperforming both generalist and molecular LLMs with a comparable number of trainable parameters.
• Physicochemical Descriptor Estimation: The model demonstrated near-perfect Pearson correlations (e.g., 0.996 for Molecular Weight, 0.999 for TPSA) and low Mean Absolute Errors, competing favorably with task-specific traditional models like Chemprop.
• Bioactivity Prediction: On the MolVision benchmark (MoleculeNet tasks), MolSight achieved the best results in 5 out of 6 tasks, including BACE, BBBP, HIV, and Tox21, surpassing both SMILES-based LLMs and image-based VLMs.

Ablation Studies:

• Removing either the MTM or MGM led to performance degradation, confirming that both topological injection and symbolic grounding are critical.
• Direct fine-tuning of the base VLM without the adaptation layer resulted in significantly lower performance, indicating that the gains are not merely due to downstream tuning.
• The model maintains strong performance even with smaller backbone sizes (4B vs. 8B), though the 8B version yields the best results.

Significance

The paper claims that MolSight demonstrates that explicitly modeling molecular topology and injecting it into visual representations is an effective path toward reliable molecular image understanding in VLMs. By bridging the gap between visual patterns and chemical graph semantics, MolSight provides a new practical foundation for a unified chemical image understanding interface. This advancement supports the transition of molecular reasoning and design from domain-specific methods toward a unified "AI chemist" paradigm, facilitating multi-modal LLM-driven drug discovery and molecular design.