Probing Materials Knowledge in LLMs: From Latent Embeddings to Reliable Predictions

This study evaluates 25 large language models on materials science tasks, revealing that while fine-tuning enhances symbolic consistency, numerical predictions remain unreliable due to inference variability and an "LLM head bottleneck" that favors embedding extraction over text output, alongside significant performance fluctuations in GPT models over time that challenge scientific reproducibility.

The Gist

Imagine you have hired a team of 25 brilliant, super-fast interns (these are the Large Language Models, or LLMs) to help you with materials science. Your goal is to see if they can act as reliable scientists: predicting how strong a material is, what color it might be, or finding facts about it in a giant library.

The researchers at MIT put these interns through a rigorous test to answer a simple question: "Can we trust these AI interns to do real science?"

Here is what they found, explained through some everyday analogies.

1. The Two Types of Tasks: "Trivia" vs. "Math"

The researchers gave the interns two very different kinds of homework:

• The Trivia Tasks (Symbolic): "What crystal system does this material belong to?" or "Fill in the blank: Titanium Dioxide is known for being ______."
• The Math Tasks (Numerical): "Predict the exact energy gap of this material" or "Calculate the dielectric constant."

The Big Discovery: The AI behaves completely differently depending on which type of homework it's doing.

For Trivia (Symbolic Tasks): The "Confused Student"

• Before Training: The raw AI models were like students who hadn't studied the textbook. They guessed wildly. If you asked them the same question 10 times, they gave 10 different, random answers. They were unreliable and inconsistent.
• After Training (Fine-tuning): When the researchers gave them specific study materials (fine-tuning), the students suddenly "got it." They stopped guessing randomly and started giving the same, correct answer every time.
• The Lesson: For facts and categories, training works wonders. It turns a confused guesser into a reliable fact-checker.

For Math (Numerical Tasks): The "Overconfident Liar"

• Before Training: This is where it gets scary. The raw AI models didn't guess randomly; they guessed precisely. If you asked them for a number, they gave a very specific number (like "4.23 eV") and they gave the exact same wrong number every single time you asked.
• The Problem: They were confidently wrong. They sounded like experts, but they were hallucinating.
• After Training: Training helped them get closer to the right number, but they still sometimes gave the same wrong answer with high confidence.
• The Lesson: You cannot trust an AI just because it sounds sure of itself. In math tasks, a "confident" answer might still be a lie.

2. The "Brain vs. Mouth" Bottleneck

The researchers did something clever. They didn't just listen to what the AI said (the text output); they looked inside the AI's "brain" (the internal data layers) while it was thinking.

• The Analogy: Imagine a student taking a test. They know the answer deep down (in their brain), but when they try to write it down on the paper (the text output), they mess up the handwriting or forget a decimal point.
• The Finding: For some properties (like "bandgap"), the AI's brain actually held the correct answer perfectly. But when the AI tried to speak it out loud, it degraded the information.
• The Solution: Instead of asking the AI to "write an essay" with the answer, the researchers found they could just "read the student's mind" (extract the internal data) and use a simple calculator to get the answer. This was often more accurate than letting the AI speak.
• The Catch: This only worked for some properties. For others (like "dielectric constant"), the AI's brain didn't even hold the right answer, so reading its mind didn't help.

3. The "Silent Update" Problem

Finally, the researchers tracked the performance of the famous "GPT" models over 18 months.

• The Analogy: Imagine you hire a tutor for your child. You test them today, and they get an A. You test them again in six months, and they get a C. You didn't change the tutor; the tutoring company just quietly swapped the tutor for a different one, or changed the textbook, without telling you.
• The Finding: The performance of these AI models fluctuated wildly (by 9% to 43%) over time. One day, a model might be great at predicting material strength; the next day, after a silent software update by the company, it might be terrible.
• The Warning: If scientists use these AI tools for long-term research, their results might not be reproducible because the "tool" they used yesterday is not the same tool they are using today.

Summary: What Should We Take Away?

1. Trust but Verify: If an AI is doing a "fact" task, training it makes it reliable. If it's doing a "math" task, be very careful—even if it sounds confident, it might be wrong.
2. Look Inside the Box: Sometimes, the best way to get an answer from an AI isn't to ask it to talk, but to peek at its internal data.
3. The "Moving Target" Risk: Using AI for science is tricky because the AI changes under the hood. Scientists need to be careful to document exactly which version of the AI they used, or their results might not hold up later.

In short, these AI models are powerful tools, but they are not magic. They have specific strengths, specific weaknesses, and they can be surprisingly unpredictable. To use them safely in science, we need to understand how they think, not just what they say.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) are increasingly adopted in materials science for tasks ranging from literature mining to property prediction. However, fundamental questions regarding their reliability, the nature of their knowledge encoding, and their behavior across different output modalities remain unresolved. Specifically, the authors address:

• How information is encoded in LLMs and whether this varies by task type (symbolic vs. numerical).
• The mechanisms behind fine-tuning success or failure.
• Whether models possess genuine understanding or merely learn statistical patterns.
• The impact of model size and the "hallucination" phenomenon on scientific reliability.
• The temporal stability of API-based models for reproducible research.

2. Methodology

The study presents a comprehensive evaluation of 25 large language models (spanning 10 architectural families, including Llama-2/3, Mistral, GPT-3.5/4 variants, and others) across four distinct materials science tasks. The evaluation covers over 200 base and fine-tuned configurations.

Tasks Evaluated:

1. Symbolic Tasks:
   • Knowledge Graph (MatKG) Link Prediction: Recasting triples (e.g., BaTiO3, has property, high dielectric constant) into natural language QA pairs.
   • Crystal System Classification: Assigning chemical formulas to one of seven crystal systems.
2. Numerical Regression Tasks:
   • Bandgap Prediction: Generating continuous bandgap values (eV) from chemical composition.
   • Dielectric Constant Prediction: Generating static dielectric tensor traces from composition.

Experimental Design:

• Fine-tuning: Open-weight models were fine-tuned using Low-Rank Adaptation (LoRA) and QLoRA. Proprietary models were fine-tuned via API.
• Consistency Measurement: Response entropy was calculated across 10 independent inference runs to quantify output consistency and confidence.
• Layer-wise Probing: To investigate internal representations, embeddings were extracted from intermediate transformer layers of fine-tuned models. Supervised probes (ridge regression, single/two-layer neural networks) were trained on these embeddings to predict properties directly, bypassing the text generation head.
• Longitudinal Study: GPT model performance was tracked over 18 months to assess temporal stability and reproducibility.

3. Key Contributions & Findings

A. Output Modality Determines Behavior (The Core Finding)

The paper identifies a fundamental asymmetry in how LLMs behave based on the output modality:

• Symbolic Tasks (Classification/Link Prediction):
  • Base Models: Exhibit high response entropy coupled with low accuracy, indicating a lack of token-level convergence (random guessing).
  • Fine-tuning: Dramatically improves accuracy (30–60% gains) and reduces entropy (by 25–99%). This suggests fine-tuning teaches models to converge on verified, consistent outputs by building distributional representations of entities.
• Numerical Tasks (Regression):
  • Base Models: Exhibit low entropy despite poor accuracy, a phenomenon termed "confident hallucination." Models generate precise, consistent numbers that are often incorrect.
  • Fine-tuning: Significantly improves prediction accuracy (RMSE reduction) but does not consistently reduce entropy. Some models become more variable in formatting while improving accuracy. This implies that for numerical tasks, low entropy is a poor indicator of reliability; models can be confidently wrong.

B. The "LLM Head Bottleneck"

Through layer-wise probing, the authors discovered that for certain properties, the model's internal representations contain more predictive information than its text output can express.

• Bandgap Prediction: Embeddings from intermediate layers achieved RMSE comparable to or better than the full fine-tuned model's text output. This indicates an "LLM head bottleneck" where the autoregressive generation process degrades the precision of the encoded numerical knowledge.
• Dielectric Constant Prediction: A stark contrast was observed. Embedding probes performed significantly worse (3× higher RMSE) than text generation. This suggests that for complex, less systematically covered properties, the knowledge is not easily accessible via static embeddings and requires the full generative process.
• Implication: The effectiveness of embedding extraction vs. text generation is property-dependent.

C. Mechanisms of Knowledge Acquisition

• Distributional Learning: In MatKG tasks, performance correlated strongly with the frequency of an entity's appearance across diverse training contexts, not with physical reasoning. Models learn statistical associations (e.g., PZT is associated with piezoelectricity because they co-occur in many prompts) rather than physical causality.
• Cross-Task Transfer: Fine-tuning on numerical tasks (bandgap) did not significantly improve symbolic tasks (crystal classification), and vice versa. This suggests that symbolic and numerical reasoning require distinct representational adaptations that do not transfer well across modalities.
• Model Scale: Once a minimum capacity threshold (~7B parameters) is met, model size has limited impact on fine-tuned performance. Smaller models (e.g., Llama-3-8B) often match or exceed larger models (e.g., Llama-2-70B) after fine-tuning.

D. Temporal Instability and Reproducibility

A longitudinal study of GPT models over 18 months revealed significant performance drift:

• Variation: Performance varied by 9–43% (in RMSE) across timepoints.
• Cause: Undocumented API updates, endpoint changes, and quantization shifts (e.g., a single update to GPT-4o caused a 43% variation).
• Implication: API-based models pose severe challenges for reproducible science. Results obtained today may not be reproducible months later without exact version strings and timestamps.

4. Significance and Implications

• Reliability Guidelines: For symbolic tasks, high entropy is a reliable signal for uncertainty. For numerical tasks, low entropy is misleading; models can be confidently incorrect.
• Efficient Prediction: For properties like bandgap, extracting embeddings and using lightweight regression may be more computationally efficient and accurate than full text generation, avoiding the "head bottleneck."
• Scientific Practice: The materials informatics community must adopt rigorous protocols for documenting model versions and evaluation dates. For long-term stability, open-weight models with frozen checkpoints are preferable to dynamic API models.
• Nature of LLM Knowledge: The study reinforces that current LLMs in materials science operate via statistical pattern matching and distributional semantics rather than genuine physical reasoning. They excel at retrieving well-attested relationships but struggle with rare combinations or complex physical derivations.

Conclusion

This work provides a critical framework for deploying LLMs in materials science, highlighting that output modality is the primary determinant of model behavior. It warns against the uncritical use of LLMs for quantitative prediction due to "confident hallucinations" and temporal instability, while offering a pathway to improve reliability through fine-tuning, embedding probing, and rigorous reproducibility standards.