Between-study variance inflates apparent accuracy in literature-compiled machine learning models of sulphate resistance in SCM-blended cement

Concrete is the most widely used building material on Earth, yet its production is a major source of global carbon emissions. To make it greener, engineers often mix in supplementary materials like fly ash, silica fume, or even agricultural waste such as rice husk ash. These additives replace some of the cement, lowering the carbon footprint while often improving the concrete's ability to resist chemical attacks. One of the most damaging threats to concrete structures is sulphate attack, which occurs when sulphate ions from soil, groundwater, or seawater seep into the material. These ions react with the cement, forming crystals that expand and crack the concrete from the inside, eventually causing it to crumble. Predicting how long a specific mix of concrete will last in such harsh conditions is vital for safety and sustainability, but traditional testing methods are slow, sometimes taking years to yield results.

In recent years, scientists have turned to machine learning to speed up these predictions. By feeding computer algorithms data from hundreds of past experiments, researchers hope to create models that can instantly estimate how strong a new concrete mix will be after exposure to sulphates. These models have been celebrated for their high accuracy, often claiming to predict outcomes with near-perfect precision. However, a new investigation suggests that this celebrated accuracy might be an illusion. The study reveals that the models are not actually learning the complex chemistry of concrete; instead, they are simply memorizing the specific laboratories that produced the data. When tested on truly new situations, these models fail completely, raising serious questions about how we evaluate the reliability of artificial intelligence in engineering.

The researchers behind this study set out to examine a dataset compiled from three separate, peer-reviewed scientific papers. These papers contained measurements of compressive strength—the force required to crush a sample—for mortar mixes containing various supplementary materials. The data covered different types of additives, varying levels of replacement, and exposure to sulphate solutions over periods ranging from a few days to over a year. In total, the team gathered 94 distinct observations. They trained a standard machine learning model on this combined data, using a technique called random splitting, which is the industry standard. In this method, the computer shuffles all the data points and sets aside a random 20 percent to test the model, while the remaining 80 percent is used for training. Because the data came from different studies, this random shuffle ensured that both the training and testing sets contained samples from all three original sources.

When the researchers evaluated the model using this standard random method, the results were spectacular. The model achieved a score indicating it could explain nearly 97 percent of the variation in the data. This level of performance is exactly what the scientific literature reports, and it suggests the model has mastered the relationship between the concrete ingredients and their durability. The model appeared to have learned that certain additives, like silica fume, generally lead to stronger concrete, while others might weaken it. It seemed to have successfully decoded the rules governing how concrete behaves under chemical attack.

However, the researchers suspected that this high score was misleading. They knew that each of the three source studies had used its own unique set of conditions: different types of cement, different mixing procedures, different curing environments, and crucially, different concentrations of sulphate solutions. In the real world, a model trained on data from one set of conditions should be able to predict the outcome for a completely new set of conditions. To test this, the researchers changed their validation method. Instead of randomly shuffling individual data points, they decided to withhold entire studies. They trained the model on two of the studies and then asked it to predict the results for the third study, which it had never seen before. They repeated this process three times, each time leaving out a different study.

The results of this stricter test were a complete collapse. When the model tried to predict the strength of concrete from a study it had not been trained on, its performance plummeted. Instead of a high score, the model produced negative scores, a mathematical indication that it was performing worse than simply guessing the average strength of the training data. In one specific case, the model was trying to predict concrete that had a measured strength between 6 and 21 megapascals, but it made errors as large as 42 megapascals. This means the model was predicting values that were more than double the actual strength of the material, or even predicting negative strength, which is physically impossible. The model had not learned the physics of concrete; it had learned to recognize the "fingerprint" of the specific laboratory that generated the data.

The investigation dug deeper to understand why this happened. The researchers broke down the total variation in the data and found that 96 percent of the differences in strength came from the differences between the studies themselves, not from the differences in the materials within a single study. One study used a specific type of cement and a low sulphate concentration, resulting in very weak concrete. Another study used a different cement and a higher concentration, resulting in very strong concrete. Because the model saw these distinct patterns repeatedly, it learned to associate the specific combination of ingredients in a study with that study's overall strength level. It was essentially learning to say, "If this data looks like it came from Study A, the strength will be low," rather than "If this concrete has silica fume, it will be strong."

This confusion was compounded by the fact that the specific additives tested were often unique to a single study. Ten out of the eleven types of supplementary materials appeared in only one of the three papers. Consequently, the model could not distinguish whether a change in strength was caused by the material or by the study it came from. Even for the one material that appeared in two studies, the average strength differed significantly between them because the underlying conditions, such as the type of sulphate salt used, were different. The model's high accuracy in the random test was an artifact of the data structure, not a reflection of genuine predictive power. It was a statistical mirage created by the way the data was grouped and tested.

The findings have profound implications for how machine learning is applied to materials science. The study demonstrates that when data is compiled from multiple sources, the standard method of randomly splitting data into training and testing sets is insufficient. It allows a model to rely on source-specific patterns rather than learning the underlying science. To truly know if a model is useful, it must be tested on data from entirely new sources, a method known as leave-one-source-out validation. Furthermore, researchers must report how much of the variation in their data comes from differences between studies versus differences within them. If the vast majority of the variation is due to the studies themselves, as it was in this case, then a high accuracy score tells us nothing about the model's ability to predict real-world durability.

Ultimately, this research serves as a necessary correction to a field that has been moving too fast. It does not suggest that machine learning cannot help engineers design better concrete, but it insists that we must be honest about what our models are actually doing. A model that looks perfect on paper but fails when faced with a new laboratory is not a tool for discovery; it is a reflection of the data's limitations. For the field to move forward, the community must adopt stricter testing standards that demand models prove they can generalize beyond the specific experiments they were trained on. Only then can we trust these digital tools to help build a more sustainable and durable future.