Regression with Large Language Models for Materials and Molecular Property Prediction

This paper demonstrates that fine-tuned LLaMA 3 models, using only composition-based input strings, can effectively predict molecular and material properties with accuracy rivaling standard machine learning models on the QM9 dataset and outperforming GPT-3.5 and GPT-4o, thereby showcasing the potential of large language models to transcend their traditional applications and tackle complex physical phenomena in materials science.

The Gist

Imagine you have a super-intelligent robot librarian named LLaMA. For years, this librarian has been famous for one thing: reading books, writing stories, translating languages, and chatting with people. It's a master of words.

But what if you asked this word-loving robot to do something completely different? What if you asked it to act like a scientist and predict the physical properties of materials—like how strong a metal is, how well a battery conducts electricity, or how much energy a molecule holds?

This paper is the story of an experiment where the researchers tried to turn LLaMA from a "word wizard" into a "science predictor."

The Big Experiment: Can a Librarian Do Math?

Usually, to predict how a material behaves, scientists use complex math models that need a very detailed "blueprint" of the material. They need to know exactly where every single atom is sitting, like a 3D map of a city.

But the researchers wanted to see if LLaMA could do it with just a name.

• The Input: Instead of a 3D map, they just gave LLaMA a text string. For a metal, it was just the chemical recipe (e.g., "Al2O3"). For a molecule, it was a text code called SMILES (which is like a shorthand recipe for how atoms are connected).
• The Task: They asked LLaMA to read that text string and guess a number (like "This metal will melt at 1200 degrees").

The Results: The "Good, Better, and Best"

Here is how LLaMA performed, using some simple analogies:

1. The "Zero-Shot" Attempt (The Untrained Librarian)
First, they asked LLaMA to guess without any training. It was a disaster. The librarian just stared blankly, said nothing, or guessed wild numbers like "-1000 degrees."

• Lesson: You can't just ask a word-expert to do science without teaching it the rules first.

2. The "Fine-Tuned" Attempt (The Trained Librarian)
Then, they "fine-tuned" LLaMA. Imagine feeding it thousands of examples: "Here is the text 'Al2O3', and here is the correct answer: 'Melting point is 2072°C'." They did this for hundreds of thousands of examples.

• The Result: LLaMA got really good! It started predicting numbers with surprising accuracy.
• The Comparison:
  • Vs. Random Guessing: LLaMA crushed it.
  • Vs. Standard Computer Models (Random Forests): LLaMA was competitive. It was often just as good as, or slightly better than, standard computer models that have been used for years.
  • Vs. The "Super-Scientists" (State-of-the-Art AI): This is where LLaMA stumbled. The absolute best AI models in the world (like PAMNet) use those detailed 3D maps of atoms. Because LLaMA only had the "text recipe" and not the "3D map," it was about 5 to 10 times less accurate than the super-scientists.

3. The "Text Format" Debate (SMILES vs. InChI)
The researchers tried giving LLaMA different types of text recipes.

• SMILES: A shorter, simpler text code.
• InChI: A longer, more complex text code.
• The Winner: LLaMA learned much faster and better with SMILES. It's like trying to learn a recipe from a short, punchy list of ingredients (SMILES) versus a long, legalistic paragraph (InChI). The shorter, simpler text was easier for the robot to understand.

4. The "Coordinates" Test (Giving the 3D Map)
They tried giving LLaMA the full 3D coordinates of the atoms (the detailed blueprint) to see if that would help.

• The Surprise: It didn't help much! Even with the full 3D map, LLaMA didn't get significantly better. This suggests that LLaMA is really good at finding patterns in text, but it might not be the best tool for processing complex 3D geometry yet.

Why Does This Matter?

Think of it like this:

• Traditional AI is like a specialized mechanic. If you want to fix a specific car engine, they are the best. But if you bring them a boat engine, they might not know what to do. They need specific tools (detailed data) for every job.
• LLaMA (The LLM) is like a universal translator. It doesn't have specialized tools, but it understands the language of the problem. If you can describe the material in text, LLaMA can learn the relationship between the description and the outcome.

The Takeaway:
This paper proves that Large Language Models (LLMs) are versatile. They aren't just for writing emails or chatting; they can actually learn to predict physical science facts just by reading text descriptions.

• Pros: You don't need to build complex 3D models or do heavy data engineering. You just need the text name of the material. It's a "plug-and-play" approach for science.
• Cons: It's currently slower and less accurate than the specialized "super-scientist" AI models that use detailed 3D data.

The Bottom Line

The researchers are saying: "We found a new way to use these AI chatbots. They aren't perfect scientists yet, but they are surprisingly good at guessing material properties just by reading their names. This opens the door for using these powerful, flexible tools in chemistry and materials science, potentially making it easier to discover new materials in the future."

Technical Summary

1. Problem Statement

The paper addresses a significant deviation from the traditional application of Large Language Models (LLMs). While LLMs are typically used for natural language processing (text generation, translation), this work investigates their ability to perform supervised regression tasks for predicting physical and chemical properties.

The core research questions are:

1. Can LLMs be fine-tuned to perform regression on material and molecular properties using only textual inputs (e.g., chemical formulas or SMILES strings)?
2. How does their performance compare to established state-of-the-art (SOTA) models and standard machine learning (ML) baselines (e.g., Random Forests, Graph Neural Networks)?
3. How do different input representations (SMILES vs. InChI vs. explicit atomic coordinates) and model architectures (LLaMA 3 vs. GPT-3.5/4o) affect regression accuracy?

A key motivation is to determine if LLMs can serve as a "general regression tool" that eliminates the need for complex, domain-specific feature engineering (featurization) required by traditional ML models.

2. Methodology

Datasets:

• Molecular Properties: The QM9 dataset was used to predict four properties: Formation Energy, HOMO, LUMO, and HOMO-LUMO gap.
  • Training sizes: 1k, 10k, and 110k molecules.
  • Test set: A fixed 10k molecules.
• Materials Properties: A diverse set of 28 properties (mechanical, thermodynamic, electronic) was analyzed.
  • Sources: Experimental data and computed data from databases like OQMD (Open Quantum Materials Database) and JARVIS-DFT.
  • Sizes: Ranging from small datasets (137 points) to massive datasets (643,916 points).

Input Representations:
The study tested three distinct input modes for fine-tuning:

1. SMILES Strings: Simplified Molecular Input Line Entry System.
2. InChI Strings: International Chemical Identifier.
3. Explicit Coordinates: Full lists of atomic coordinates and element types (XYZ format).
   Note: For materials properties, only composition strings (e.g., "Al2O3") were used.

Model Architecture & Training:

• Primary Model: LLaMA 3 (8B parameters) fine-tuned using LoRA (Low-Rank Adaptation) with 4-bit quantization.
• Training Objective: The models were trained solely on generative cross-entropy loss. Crucially, the regression target (a numerical value) was treated as a text token. The authors did not use a custom regression head or minimize Mean Squared Error (MSE) directly.
• Baselines:
  • Random Forest (RF): Trained on elemental descriptors (Jacobs et al. method).
  • PAMNet: A state-of-the-art Physics-Aware Multiplex Graph Neural Network (using full structural coordinates).
  • AtomGPT: A previous LLM-based method using a modified regression head.
  • GPT-3.5 and GPT-4o: Fine-tuned via OpenAI API for comparison.

3. Key Contributions

1. Feasibility of Text-to-Regression: Demonstrated that LLMs can successfully learn to predict numerical physical properties from purely textual inputs (composition/SMILES) by optimizing only for generative loss, without explicit regression heads.
2. Benchmarking Against SOTA: Provided a comprehensive comparison showing that while LLMs are competitive with standard ML models (Random Forest), they currently lag behind SOTA deep learning models that utilize explicit structural data.
3. Input Sensitivity Analysis: Quantified the impact of input representation, finding that SMILES strings outperform InChI strings by ~15-20% in error reduction, and that adding explicit atomic coordinates yields only marginal improvements over SMILES for LLMs.
4. Model Selection: Showed that LLaMA 3 outperforms GPT-3.5 and GPT-4o in this specific regression context, likely due to the flexibility of open-source fine-tuning hyperparameters.

4. Key Results

Molecular Properties (QM9 Dataset):

• Performance: LLaMA 3 achieved a Mean Absolute Error (MAE) of 0.100 eV for formation energy (trained on 110k points).
• Comparison to SOTA: This is ~17x worse than PAMNet (MAE 0.0059 eV), which uses full atomic coordinates.
• Comparison to Baselines: LLaMA 3 performed comparably to neural networks trained on SMILES-derived features (Pinheiro et al.), suggesting the gap with PAMNet is largely due to the lack of explicit structural data in the LLM input.
• Zero-Shot Failure: Without fine-tuning, LLaMA 3 failed completely (MAE ~59 eV), highlighting the necessity of fine-tuning.
• Input Type: SMILES strings yielded ~25% lower errors than InChI strings. Explicit coordinates did not significantly improve results over SMILES in the initial tests.

Materials Properties (28 Datasets):

• vs. Random Forest: LLaMA 3 performed better than Random Forest in 8/23 cases, equally in 7 cases, and worse in 8 cases.
• Average Error: The average RMSE/σ for LLaMA 3 was 8.6% higher than Random Forest.
• Large Scale (OQMD): On the OQMD formation energy dataset (634k points), LLaMA 3 achieved an MAE of 0.054 eV. This is comparable to the fully connected neural network ElemNet (0.055 eV) but worse than RoosT (0.024 eV), which uses stoichiometry-based representation learning.
• vs. AtomGPT: LLaMA 3 slightly outperformed AtomGPT on two properties and underperformed on two others, despite using a simpler training approach (direct generative loss vs. modified regression head).

Model Comparisons:

• LLaMA 3 vs. GPT: LLaMA 3 (MAE 0.100 eV) outperformed GPT-4o-mini (MAE 0.154 eV) on QM9 formation energy. The authors attribute this to the limited hyperparameter tuning available for closed-source GPT models.

5. Significance and Conclusion

• Versatility of LLMs: The study proves that LLMs can transcend NLP to model complex physical phenomena, acting as a "featurization engine" that internally learns representations from minimal input (composition/SMILES).
• Trade-offs:
  • Pros: LLMs require no manual feature engineering and can handle diverse property types with a single architecture. They are competitive with standard ML (Random Forest) on small-to-medium datasets.
  • Cons: They are computationally expensive to train (hours of GPU time vs. seconds for Random Forest) and currently lack the precision of SOTA models that utilize explicit structural data (coordinates).
• Future Directions: The authors suggest that future improvements could come from:
  • Better integration of structural information (beyond simple XYZ files).
  • Exploring other string representations (e.g., SELFIES, DeepSMILES).
  • Multitask learning and transfer learning between properties.
  • Investigating larger model sizes (e.g., LLaMA 70B).

In summary, this work establishes LLMs as a viable, flexible tool for materials property prediction, particularly in scenarios where featurization is difficult, though they are not yet a replacement for specialized, structure-aware deep learning models in terms of raw accuracy.