Scale-Dependent Input Representation and Confidence Estimation for LLMs in Materials Property Prediction

This study demonstrates that optimal input representations for materials property prediction depend on LLM scale, with compact formats suiting smaller models and detailed descriptions benefiting larger ones, while establishing mean negative log-likelihood as an effective, training-free confidence metric for fine-tuned models.

The Gist

Imagine you are trying to teach a computer to guess the properties of a new material, like how much energy it takes to build it or how well it conducts electricity. This paper is like a guidebook for two different-sized "brains" (AI models) on how to best understand the instructions you give them.

Here is the story of what the researchers found, broken down into simple concepts:

1. The Two Brains: A Toddler vs. A Professor

The researchers tested two versions of an AI called "Llama":

• The 1B Model (The Toddler): A smaller, simpler brain.
• The 8B Model (The Professor): A larger, more complex brain with more knowledge.

They wanted to see if the size of the brain changed how it should be taught. They gave these models five different ways to describe a material (like a crystal):

1. The Recipe Card: Just the list of ingredients (Chemical Composition).
2. The Headline: A short summary including the ingredients and the material's "shape" or symmetry (Crystal Summary).
3. The Local Tour: A description of how the atoms are hugging each other nearby (Local Environment).
4. The Full Novel: A long, detailed story describing the entire structure (Full Description).
5. The Blueprints: A raw, technical file full of numbers and coordinates (CIF).

2. The "Short vs. Long" Lesson

The biggest discovery was that one size does not fit all.

• For the Toddler (1B Model): It got confused by long stories. When you gave it the "Full Novel" or the complex "Blueprints," it stumbled. It worked best when you gave it the Recipe Card or the Headline. It needed short, punchy facts to get the job done right.
• For the Professor (8B Model): This brain loved the details. When you gave it the Full Novel, it actually performed better than with the short summaries. It could read the long, complex descriptions and pull out the subtle clues it needed to make a great guess. However, even the Professor struggled a bit with the raw "Blueprints" (the technical files), suggesting that natural language (words) is still easier for these AI brains to understand than raw code.

The Golden Rule: If you have a small AI, keep your instructions short. If you have a big AI, you can give it a detailed story.

3. The Magic of "Symmetry"

One specific ingredient in the instructions turned out to be a superpower for both the Toddler and the Professor: Symmetry.

Imagine you have two different shapes made of the same Lego bricks. If you only tell the AI "It's made of red and blue bricks," the AI can't tell the shapes apart. But if you add the "Headline" which says, "It's a square shape," the AI suddenly knows the difference. The paper found that including information about the material's symmetry (its shape/group) helped both models guess the properties much more accurately than just listing the ingredients.

4. The "Confidence Meter" (How to know if the AI is guessing)

The second big question was: How do we know if the AI is confident in its answer, or just making it up?

In the world of AI, there is a number called NLL (Negative Log-Likelihood). Think of this as the AI's internal "confidence meter."

• Low NLL: The AI is very sure of its answer.
• High NLL: The AI is unsure or guessing.

The Catch:

• Before Training: When the AI was just a "base" model (not yet taught about materials), this confidence meter was broken. It would say "I'm super sure!" even when it was completely wrong.
• After Training: Once they "fine-tuned" (taught) the models using a special method called LoRA, the meter started working! They found a clear pattern: When the AI's confidence meter was high (low NLL), its answers were usually correct.

This means that after training, you can look at the AI's internal confidence score to decide whether to trust its prediction. If the score is low (high uncertainty), you can ignore that answer and save yourself from a bad guess.

5. The Trade-off: Speed vs. Accuracy

The paper also noted a practical downside. While these AI models are smart and flexible, they are slow.

• A traditional, specialized computer program (like a graph neural network) could check 10,000 materials in about one minute.
• These AI models took several hours to do the same job.

Summary

This paper teaches us that when using AI to predict material properties:

1. Match the input to the model: Don't give a small AI a long story; give it a summary. Give a big AI the full story.
2. Include symmetry: Telling the AI about the material's shape helps it guess better.
3. Train first, then trust: You must teach the AI about materials before you can trust its "confidence meter." Once trained, that meter is a great tool to filter out bad guesses.

The researchers didn't claim this is ready to replace all current tools immediately (due to the slow speed), but they showed that with the right setup, these flexible AI models can be very effective and self-aware tools for scientists.

Technical Summary

Technical Summary: Scale-Dependent Input Representation and Confidence Estimation for LLMs in Materials Property Prediction

Problem Statement

While Large Language Models (LLMs) are increasingly applied to materials science for tasks such as property prediction, two critical challenges remain unresolved:

1. Input Representation vs. Model Scale: It is unclear how the optimal input representation (e.g., chemical composition, natural language descriptions, or structured files) depends on the scale of the LLM and its fine-tuning status. Prior studies utilize diverse formats and model sizes, making systematic comparison difficult.
2. Confidence Estimation: Reliable methods for assessing the confidence of LLM-generated property predictions are lacking. Existing uncertainty quantification (UQ) methods for graph neural networks often require additional modeling overhead. While LLMs naturally provide token-level probabilities (Negative Log-Likelihood, NLL), their applicability as a confidence metric for numerical property prediction remains unverified.

Methodology

The study conducts systematic experiments using the LLM4Mat-Bench dataset (derived from the Materials Project), focusing on two target properties: formation energy per atom and bandgap.

• Models: Two Llama models of different scales were utilized: Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct. Both were evaluated in their base (pre-trained only) and fine-tuned states.
• Fine-tuning: Models were fine-tuned using Low-Rank Adaptation (LoRA) applied to query and value projection layers (rank $r=32$, scaling factor $\alpha=64$). Training was conducted for 6 epochs with a learning rate of $1 \times 10^{-4}$.
• Input Representations: Five distinct input modalities were constructed for each sample:
  1. Composition: Chemical formula only.
  2. Crystal Summary: The leading sentence of a natural-language description (includes composition and space-group).
  3. Local Environment: The remaining descriptive text excluding the summary sentence.
  4. Full Description: The complete natural-language text.
  5. CIF: Raw Crystallographic Information File strings.
• Evaluation Metrics:
  • Accuracy: Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) between predicted and true values.
  • Confidence: The Mean Negative Log-Likelihood (Mean NLL) of tokens corresponding to the predicted numerical values. Specifically, the study focuses on the integer part of the numerical string to avoid noise from fractional digit tokenization.
  • Filtering: A "NLL filtering" strategy was tested, where predictions with Mean NLL above a certain threshold are discarded to improve the reliability of the remaining set.

Key Results

1. Scale-Dependent Input Representation

The optimal input representation is strongly dependent on the model scale:

• 1B Model (Small Scale): Performs best with compact representations (Composition and Crystal Summary). As input length and complexity increase (e.g., Full Description, Local Environment), the Mean Absolute Error (MAE) increases, and training instability (variance across seeds) rises. The 1B model struggles to map long-form text or structured CIF data to precise physical properties.
• 8B Model (Large Scale): Demonstrates robustness to detailed inputs. For formation energy, the 8B model achieves its lowest MAE with the Full Description, leveraging its pre-trained natural language understanding to extract nuanced structural features.
• Symmetry Information: Across both model scales, the Crystal Summary (which includes space-group information) consistently outperforms Composition-only inputs. This indicates that symmetry descriptors act as robust features that help distinguish polymorphs and activate crystallographic knowledge embedded in the LLM.
• CIF Performance: While the 8B model can interpret CIF data, natural-language descriptions generally yield better accuracy, suggesting LLM internal representations are more aligned with natural language than raw coordinate data.

2. Confidence Estimation via Mean NLL

• Base Models: No clear correlation exists between Mean NLL and prediction error. Large errors occur even at low NLL values, indicating that pre-trained probabilities reflect biases rather than material-property relationships.
• Fine-Tuned Models: A consistent trend emerges where lower Mean NLL corresponds to smaller prediction errors. This correlation holds across different model scales and input representations.
• NLL Filtering: By applying a threshold to the Mean NLL (discarding high-NLL predictions), the MAE of the retained predictions decreases significantly below the baseline. This demonstrates that Mean NLL serves as a practical, training-free confidence indicator for fine-tuned models.
• Token Scope: The study found that restricting the NLL calculation to the integer part of the numerical value is more reliable than including fractional digits, as the latter introduces noise due to tokenization ambiguity.

Key Contributions

1. Systematic Analysis of Scale and Representation: The study establishes that input design must be tailored to model capacity. Compact inputs are optimal for smaller models (1B), while larger models (8B) benefit from detailed natural-language descriptions.
2. Validation of Symmetry Features: It demonstrates that including space-group information in input summaries is a critical factor for improving prediction accuracy across model scales.
3. Confidence Indicator for LLMs: The paper provides evidence that Mean NLL of numerical tokens can serve as an effective confidence metric for materials property prediction, but only after task-specific fine-tuning. This offers a computationally efficient alternative to complex UQ methods.

Significance and Limitations

The authors claim that these findings provide practical guidance for designing input representations and assessing prediction reliability in LLM-based materials informatics. The ability to filter predictions based on internal confidence scores (Mean NLL) allows for more reliable deployment without additional training overhead.

Limitations acknowledged by the authors:

• Model Scope: The analysis is limited to 1B and 8B models; generalization to larger scales (e.g., 70B) requires further investigation.
• Property Scope: Results are specific to formation energy and bandgap; other properties may behave differently.
• Computational Cost: LLM inference is significantly slower (hours vs. seconds for GNNs like CGCNN) and requires substantial GPU memory, limiting immediate scalability for high-throughput screening compared to specialized models.
• Architecture Specificity: Findings are specific to the Llama 3 series; validation on other architectures is needed.
• Exploratory Nature: The confidence thresholding is based on test-set observations; practical deployment requires threshold selection on a held-out validation set.

The study concludes that while LLMs may not yet surpass specialized Graph Neural Networks (GNNs) in raw accuracy for specific tasks, their flexibility in input design and potential for multi-task application without task-specific architectures represent significant practical advantages.