MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition

Technical Summary: MarkushGlyph and OCSRGlyph

Problem Statement
Chemical structures in patents and scientific literature are predominantly communicated as images, which are readable by chemists but difficult to parse for programmatic use (e.g., database indexing or machine learning training). The task of converting these images into machine-readable line notations (typically SMILES or CXSMILES) is known as Optical Chemical Structure Recognition (OCSR) for single molecules and Markush structure translation for families of compounds. While OCSR has seen significant maturity, Markush structure parsing remains challenging due to the need to resolve variable sites, constraints, and diverse drawing conventions across patent offices. Furthermore, existing evaluation metrics for Markush structures often fail to penalize incorrect feature additions or label transpositions, leading to inflated accuracy scores.

Methodology
The authors frame both OCSR and Markush translation as image-to-text translation problems, training single models rather than relying on multi-stage pipelines that separately process text, layout, and images.

1. OCSRGlyph (Single Molecule Recognition):

   • Architecture: A standard image-to-sequence model using a Swin-B Transformer encoder (384-pixel resolution, ImageNet-initialized) and a six-layer Transformer decoder.
   • Training Strategy: The model is trained on a curated mixture of PubChem-1M and USPTO-680K datasets. Crucially, the authors address the specific challenge of stereochemistry by oversampling chiral molecules (Stereo-200K and an adjacent-ring subset) during training. This improves stereochemical accuracy without degrading performance on non-chiral molecules.
   • Output: Generates bare canonical SMILES strings.

2. MarkushGlyph (Markush Structure Recognition):

   • Architecture: A vision-language model (VLM) based on Qwen3.5-2B-Base, utilizing an early-fusion approach where the cropped image and a text prompt are processed jointly by a single network. It employs Low-Rank Adaptation (LoRA) for efficient fine-tuning.
   • Input/Output: The model accepts the entire structure image (including printed text, R-group labels, and brackets) without a separate OCR front-end. It outputs a structured cxsmiles_opt record (an optimized CXSMILES format where labels are inline with atoms) wrapped in XML tags, including the core structure and any substituent definitions.
   • Combined Training: To handle both single molecules and Markush structures without prior knowledge of the input type, the model is trained on a combined dataset where ordinary molecules are treated as a special case of Markush records (with empty variable fields).

Key Contributions

• OCSRGlyph: A state-of-the-art OCSR model that achieves 93.8% exact-match accuracy on the USPTO benchmark by specifically addressing stereochemistry through data curation.
• MarkushGlyph: A single-stage vision-language model that outperforms prior multi-stage pipelines (like MarkushGrapher-2) on all three standard Markush benchmarks (IP5-M, M2S, USPTO-M). It eliminates the need for separate text extraction and layout parsing stages.
• Strict Parsed-Graph Equality Metric: The authors introduce a new evaluation metric that addresses failure modes in existing metrics. Unlike prior methods that check for the presence of required features (allowing extra or incorrect features), this metric requires strict graph isomorphism between the prediction and the reference, preserving R-group assignments, attachment positions, repeat units, and fragment correspondence.
• Unified Approach: Demonstrating that OCSR can be treated as a special case of Markush translation, allowing a single model to parse both types of chemical structures effectively.

Results

• OCSR Performance: OCSRGlyph achieves 93.8% canonical exact-match accuracy on the USPTO benchmark (5,719 images), surpassing the previous state-of-the-art (MolSight) by 1.8 percentage points. It also achieves 93.9% accuracy under the "chirality-kept" convention.
• Markush Performance: MarkushGlyph outperforms MarkushGrapher-2 and other baselines across all three benchmarks under both the standard MarkushGrapher-2 score and the new strict parsed-graph equality metric.
  • On IP5-M (878 examples), MarkushGlyph achieves 60.6% (majority voting) under the standard score and 54.7% under strict equality.
  • On M2S (103 examples), it reaches 62.1% (standard) and 64.1% (strict).
  • On USPTO-Markush (74 examples), it reaches 63.5% (standard) and 56.8% (strict).
• Ablation Studies: The authors show that adding OCSR samples to the Markush training set significantly improves OCSR performance without substantially degrading Markush recognition capabilities.

Significance and Claims
The paper claims that these results represent a step toward a unified model capable of translating both single molecules and complex Markush families from images. The authors emphasize that high performance is achieved through relatively simple strategies: leveraging pre-trained backbones, single-stage training, and careful data sampling.

The work highlights two specific limitations:

1. MarkushGlyph's OCSR performance, while strong, does not yet match the specialized OCSRGlyph model, suggesting a gap remains in unifying state-of-the-art performance for both tasks in a single architecture.
2. Markush structure recognition performance still lags significantly behind OCSR performance, indicating that the complexity of variable groups and constraints remains a challenging frontier.

The authors conclude that their work demonstrates the efficacy of simple, scalable methods and inference-time scaling (e.g., majority voting) in closing the gap between current capabilities and the requirements for robust chemical structure recognition in patent and literature analysis.