Chemically Meaningful Textualization Enables Explainable Validation of Metal-Organic Frameworks by Large Language Models

Technical Summary: Chemically Meaningful Textualization Enables Explainable Validation of Metal-Organic Frameworks by Large Language Models

Problem Statement
Computation-ready Metal-Organic Framework (MOF) databases are critical for high-throughput screening and machine learning (ML) in materials science. However, many experimentally reported crystal structures contain defects such as disorder, missing atoms, charge imbalances, or chemically unreasonable coordination environments. These structural inconsistencies severely distort computed properties (e.g., electronic energy, band gaps, and gas adsorption capacities), compromising the reliability of simulations. While existing validation strategies exist, they suffer from significant limitations: rule-based approaches often lack precision or require restricted access to commercial databases, and graph-based ML models typically function as "black boxes" that identify unreasonable structures without offering interpretable diagnoses of the underlying errors.

Methodology
The authors investigate whether Large Language Models (LLMs) can serve as interpretable validators for MOF structures by transforming crystallographic information into "chemically meaningful text." The study proceeds through several key methodological steps:

1. Impact Assessment: The authors first demonstrated that structural defects in MOFs lead to substantial deviations in predicted electronic properties (via PACMAN) and gas adsorption simulations (via Grand Canonical Monte Carlo). They showed that corrected structures reproduce experimental adsorption isotherms more faithfully than raw, defective structures.
2. Descriptor Benchmarking: Nine different descriptors were benchmarked to determine the optimal way to represent MOF structures for LLMs. These descriptors spanned three categories:
   • Non-text: Chemical elements and numbers only.
   • Semi-text: Combinations of strings and structural data.
   • Full-text: Natural language descriptions of the structure.
     Specific descriptors included local_env, robocry, mof2text, cif_p1, and others.
3. Model Training and Evaluation: The study utilized four pre-trained LLMs (Llama3-8B, DeepSeek-7B, Gemma3-4B, Qwen3-4B). The models were evaluated in both pre-trained (PT) and fine-tuned (FT) states using four metrics: Accuracy, Precision, Recall, and F1 score.
4. Explainability and Diagnosis: Beyond binary classification (reasonable vs. unreasonable), the authors tested the ability of fine-tuned LLMs to generate diagnostic rationales for errors. They analyzed the embedding space using t-SNE to see if descriptors could separate reasonable and unreasonable structures. Finally, they evaluated the models' ability to predict specific error categories (missing hydrogen "h", disorder, charge imbalance, and "other") on a manually curated dataset.

Key Contributions and Results

• The Necessity of Chemically Informed Textualization: The study found that the success of LLM-based validation does not depend solely on the quantity of structural information provided. Instead, it depends on whether the information is organized into a "linguistically learnable" and chemically meaningful form. Simple compositional inputs or raw geometric transcriptions (like cif_p1) performed poorly or were computationally inefficient due to excessive token lengths.
• Superiority of Specialized Descriptors: Fine-tuned LLMs using specialized full-text descriptors, particularly mof2text and robocry, achieved performance comparable to state-of-the-art graph-based models (e.g., PMTransformer, CGCNN) and significantly outperformed geometry-based heuristics (e.g., Chen-Manz, MOFChecker). The mof2text descriptor, which encodes topology, charge information, and building units while avoiding redundant atom-by-atom descriptions, yielded the highest accuracy.
• Explainable Validation: Unlike traditional classifiers, the fine-tuned LLMs could generate specific diagnostic rationales. When provided with mof2text inputs, models like Qwen3 successfully identified defects related to "bonding," "connectivity," and "charge states." The t-SNE analysis revealed that chemically meaningful descriptors created embedding spaces where reasonable and unreasonable structures were clearly separated, and error types formed discernible clusters.
• Error Categorization: On a manually labeled four-class dataset, fine-tuned LLMs (specifically FT-Llama3-R) correctly identified error types for approximately 70% of unreasonable MOFs, with only ~12% incorrect predictions. The models demonstrated an ability to map structural defects to chemically plausible categories, capturing the broader chemical consequences of errors (e.g., linking missing hydrogen atoms to coordination and charge anomalies).

Significance and Claims
The paper claims to establish "chemically informed textualization" as the critical enabling step for transforming general-purpose LLMs into practical, explainable tools for curating MOF databases. The authors assert that:

1. LLMs can move beyond black-box classification to provide interpretable rationales for structural defects, bridging the gap between prediction and explanation.
2. The representation of data is more critical than the model architecture itself; without chemically meaningful textualization, LLMs cannot effectively internalize crystallographic validity.
3. This approach offers a viable route for AI-assisted curation of computation-ready MOF databases, potentially facilitating future workflows that combine validation, rationale generation, and automated refinement.

The authors remain modest regarding the current limitations, noting that the four-class error framework is a simplified approximation and that the models cannot yet precisely localize erroneous atoms or fully specify root causes for all defect types. However, they position this work as a foundational step toward more general, autonomous refinement workflows in reticular chemistry.