🤖 machine learning

Mat-Pref: Verifiable-Reward Training Improves Compositional Reasoning in Inorganic Materials

This paper introduces Mat-Pref, a verifiable-reward benchmark for inorganic materials, and demonstrates that a two-stage training pipeline combining supervised fine-tuning with Group Relative Policy Optimization (GRPO) significantly outperforms larger zero-shot models by enhancing compositional reasoning and generalization to unseen structure families and properties.

Original authors: Sarrah R. Mikhail Leung, Taehan Kim, Jeongbin Park

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sarrah R. Mikhail Leung, Taehan Kim, Jeongbin Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced chef how to cook a perfect meal. You have a massive library of recipes (the training data), but the chef keeps guessing the wrong ingredients or getting lost in the instructions.

This paper, MAT-PREF, is about teaching a specific type of "chef" (an Artificial Intelligence) how to solve complex puzzles involving inorganic materials—think of these as the building blocks for new batteries, solar panels, or computer chips.

Here is the story of what they did, explained simply:

1. The Problem: The Chef Can't "Reason," They Just Guess

The researchers wanted to see if big AI models could figure out the best way to swap one ingredient (an atom) in a crystal structure for another to make it stronger or more efficient.

They tested four of the world's smartest AI chefs (models with 70 to 671 billion "brain cells"). Even though these models are huge, they only got about 33% to 54% of the answers right.

The Analogy: It's like giving a genius a multiple-choice math test, but they keep picking the wrong answer because they are just guessing based on how the words look, rather than actually doing the math. They haven't learned the logic of chemistry yet.

2. The Solution: A New "Training Gym" (MAT-PREF)

To fix this, the team built a new training ground called MAT-PREF.

The Source: They used a massive, trusted database of chemical facts (Materials Project) where every answer is verified by a super-accurate computer simulation (called DFT). It's like having a teacher who knows the exact right answer for every single question.
The Test: They created over 10,000 questions. Some were easy (similar to what the AI saw in training), some were hard (totally new crystal shapes), and some were tricky (using the same shapes but asking about a different property, like "how much light it blocks" instead of "how stable it is").

3. The Training Method: Two Steps to Success

The researchers didn't just feed the AI more data. They used a two-step training process:

Step 1: Supervised Fine-Tuning (SFT) – "Learning the Rules"
First, they taught the AI how to think like a chemist. They gave it examples of questions and the correct reasoning steps (e.g., "Check the size of the atom," "Check the electrical charge").
- Result: The AI got much better at following the format and understanding the logic, jumping from near-random guessing to about 45% accuracy. But it was still inconsistent.
Step 2: GRPO (Group Relative Policy Optimization) – "Learning from Mistakes"
This is the secret sauce. Imagine the AI tries to answer a question 8 times.
- If it gets the answer right, it gets a "treat" (a reward).
- If it gets it wrong, it gets a "no."
- The AI then compares its 8 attempts. It learns: "Hey, the answer I picked in attempt #3 was right, but the one in #7 was wrong. I should stop picking #7."
- Result: This process forces the AI to stop guessing and start committing to the correct logic.

4. The Results: A Small Chef Beats the Giants

After this two-step training, they took a relatively small AI model (Qwen3-8B) and tested it again.

The Shock: This small, trained model scored 65% to 71% accuracy.
The Comparison: It beat the massive, untrained "giant" models (which had 30 times more brain power) by a huge margin (over 20 percentage points).
The Cost: They did all this training for less than $50.

5. Why It Matters: Consistency is Key

The paper found something fascinating about how the AI improved.

Before: The AI knew the right answer sometimes, but it was like a nervous student who knew the answer but was afraid to raise their hand. If you asked the same question with slightly different wording (distractors), the AI would flip-flop between right and wrong.
After: The training made the AI confident. It didn't just find the right answer; it learned to stick with it.
- The Analogy: Think of a student taking a test. Before, they might circle "A" on one version of the question and "B" on another, even though the logic is the same. After training, they circle "A" every single time because they truly understand the concept.

Summary

The paper shows that size isn't everything. You don't need the biggest, most expensive AI to solve complex science problems. Instead, you need a smart training method that uses verified facts to teach the AI how to reason logically and confidently. By using this new "gym" (MAT-PREF) and the two-step training, a small AI learned to outperform the giants at figuring out how to build better materials.

Technical Summary: MAT-PREF: Verifiable-Reward Training Improves Compositional Reasoning in Inorganic Materials

Problem Statement

Reinforcement learning from verifiable rewards (RLVR) has driven significant progress in mathematical and code reasoning, yet its application to scientific domains remains under-evaluated regarding the nature of generalization. Existing benchmarks in materials science often fail to decompose whether performance gains stem from structural transfer, property transfer, or mere memorization. Specifically, the task of ionic substitution in inorganic crystals requires compositional reasoning that jointly considers oxidation states, coordination environments, ionic radii, and electronic-structure effects. Current large language models (LLMs), even those with hundreds of billions of parameters, struggle to reliably compose these factors, performing near random chance on such tasks. Furthermore, it is unclear whether verifiable reward training can improve reasoning in domains where correctness depends on Density Functional Theory (DFT) property rankings rather than simple pattern matching.

Methodology

The MAT-PREF Benchmark

The authors introduce MAT-PREF, a benchmark comprising 10,837 multiple-choice questions on ionic substitution across 11 inorganic structure families. The data is grounded in DFT calculations from the Materials Project.

Construction: Entries are filtered for stability ( $E_{hull} \le 0.1$ eV/atom), classified into structure families, assigned oxidation states, and validated at the site level. Questions are generated by removing a substituted element to create a "template group," ensuring that property differences arise solely from the substituted element.
Evaluation Splits: The benchmark isolates three orthogonal axes of generalization:
1. IID (In-Distribution): Novel hosts within familiar training structure families.
2. OOD-host (Out-of-Distribution Hosts): Entirely held-out structure families (Garnet, Halide Perovskite, NASICON).
3. OOD-property: Held-out target properties (band gap) for hosts seen during training only via a different property (formation energy).
Scoring: To ensure robustness against distractor sensitivity, accuracy is measured at the logical question level (majority vote across multiple distractor permutations of the same underlying chemical question).

Training Pipeline

The authors employ a two-stage training pipeline on 8B-parameter models (Qwen3-8B and Llama-3.1-8B-Instruct):

Supervised Fine-Tuning (SFT): A reasoning trace is generated for each training question using DeepSeek-R1, instructed to reason from chemical principles (ionic radius, coordination, etc.) without revealing numerical DFT values. The model is fine-tuned on these (question, trace) pairs to bootstrap format and chemical vocabulary.
Group Relative Policy Optimization (GRPO): Starting from the SFT checkpoint, the model is trained using GRPO with a binary verifiable reward. The reward is 1 if the extracted answer matches the DFT-backed ground truth and 0 otherwise. This stage refines the policy to maximize the probability of the correct answer without requiring explicit process rewards.

Key Contributions

MAT-PREF Benchmark: A DFT-grounded benchmark with three orthogonal evaluation axes and a scoring protocol that penalizes inconsistent reasoning across distractor permutations.
Performance Gains via Verifiable Rewards: Demonstration that a two-stage SFT+GRPO pipeline lifts an 8B model to 65.2% accuracy on IID and 71.6% on OOD-host, exceeding the strongest zero-shot baseline (Qwen3-235B-Instruct) by over 20 percentage points on structural generalization splits.
Mechanistic Insight into Consistency: Evidence that GRPO reshapes the policy distribution so that correct answers become the modal response rather than merely reachable. This is quantified by a reduction in the gap between lenient scoring (at least one permutation correct) and strict scoring (all permutations correct) from 24.0pp to 14.3pp. Logit lens analysis reveals a ~20pp advantage in "answer crystallization" at the critical decision layer (layer 24) for GRPO models compared to SFT.

Results

Scale vs. Reasoning: Four zero-shot frontier models (70–671B parameters) remain in the 33–54% accuracy range across all splits, confirming that model scale alone does not resolve the compositional reasoning required for ionic substitution.
Training Efficacy: The SFT+GRPO pipeline on Qwen3-8B achieves 65.2% (IID), 71.6% (OOD-host), and 60.0% (OOD-property). These results are replicated on Llama-3.1-8B with identical hyperparameters, confirming the findings are not architecture-specific.
Generalization Patterns:
- Structural Transfer: The model generalizes well to unseen families, with NASICON achieving the highest OOD accuracy (82.1%).
- Property Transfer: Cross-property transfer (training on formation energy, testing on band gap) yields 60.0% accuracy, significantly above random baselines but with a smaller margin over zero-shot models, suggesting frontier pretraining captures some cross-property reasoning.
- Bottlenecks: Electronic structure reasoning (band gap) remains a primary bottleneck, lagging behind thermodynamic reasoning (formation energy) by over 20 percentage points on IID splits.
Consistency: GRPO significantly improves consistency across distractor permutations, reducing the lenient-strict scoring gap and demonstrating sharper internal commitment to the correct answer.

Significance and Claims

The paper claims that verifiable-reward training is a viable and effective method for improving compositional scientific reasoning in materials science, even when the training data is limited to 8B-parameter models. The results suggest that:

Format and Reasoning are Separable: SFT resolves format and vocabulary issues, while GRPO refines the reasoning policy against verifiable physical laws.
Generalization is Achievable: Models can generalize to entirely new structure families and transfer reasoning across different physical properties, provided the training signal is grounded in first-principles calculations.
Consistency is a Key Metric: The ability to produce consistent answers across surface-level perturbations (distractor permutations) is a distinct and measurable benefit of RLVR, indicating a shift from surface cue reliance to deeper chemical reasoning.

The authors remain modest regarding the scope, noting that MAT-PREF tests reasoning within a DFT-consistent framework rather than real-world synthesis feasibility, and that electronic-structure reasoning remains a limitation requiring future work, such as process reward models.