MolSanity: A Reliability Audit for Feature Attributions on Molecular Graph Neural Networks
MolSanity introduces a comprehensive reliability audit framework for molecular graph neural network feature attributions, revealing that faithfulness to the model does not correlate with chemical correctness or ground-truth localization, particularly under scaffold shifts, thereby demonstrating that explanation reliability must be evaluated on a case-by-case basis rather than assumed from faithfulness metrics alone.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the high-stakes world of drug discovery, scientists often rely on artificial intelligence to predict how a new chemical compound will behave in the human body. These systems, known as graph neural networks, treat molecules as collections of atoms connected by bonds, much like a map of cities and roads. When the AI makes a prediction, such as flagging a compound as potentially toxic, it is standard practice to ask the system to explain its reasoning. The AI responds by highlighting specific atoms on the molecule, creating a heat map that suggests which parts of the chemical structure were most important for the decision. Chemists trust these maps to guide their work, assuming the highlighted atoms are the true chemical drivers of the result.
However, a new study reveals a troubling disconnect between what these AI systems say they did and what they actually did. The research introduces a rigorous audit called MolSanity, designed to test whether these explanations are reliable. The central finding is that an explanation can be perfectly consistent with the AI's internal logic while being completely wrong about the chemistry. In other words, the system can be honest about its own thought process, yet that process might be based on a misunderstanding of the molecule. This distinction matters because if a scientist relies on a misleading heat map, they might discard a promising drug or, worse, pursue a dangerous one, believing the AI has identified the correct chemical culprit when it has not.
The researchers set out to investigate this gap by wrapping a wide array of existing AI tools and testing them against a massive collection of molecular data. They did not invent a new way to generate explanations; instead, they took the standard tools used by the scientific community and subjected them to a strict reliability audit. This audit looked at every possible combination of dataset, AI architecture, explanation method, and data split. They measured two distinct things: faithfulness and correctness. Faithfulness asks whether the explanation accurately reflects the model's internal calculations. If the model says "I looked at this atom," does the explanation point to that atom? Correctness asks whether the explanation identifies the actual chemical feature that determines the property. If the model is predicting toxicity, does the explanation point to the toxic part of the molecule?
The study found that these two qualities are not linked. A method can be highly faithful, meaning it perfectly describes the model's behavior, and yet be completely incorrect, pointing to the wrong chemical features. In fact, across thirty different tests, a ranking system that chose the best explanation method based solely on faithfulness picked the wrong method twenty-six times. The researchers tested this under two conditions: when the AI was tested on molecules similar to those it was trained on, and when it was tested on entirely new chemical structures, a scenario known as a scaffold shift that mimics real-world drug discovery. The failure was not limited to the difficult new cases; it happened just as often in the familiar ones. In one specific case involving a well-known dataset, the most faithful method pointed to atoms that were the exact opposite of the true toxic group, while the method that correctly identified the toxic group was ranked as less faithful.
This lack of connection means that there is no single "best" explanation method that works for every situation. The reliability of an explanation depends on the specific combination of the dataset, the AI model, and the method used to generate the explanation. The researchers discovered that while the ranking of explanation methods remained relatively stable across different types of molecules, the relationship between faithfulness and correctness varied wildly. On some datasets, the two were positively linked; on others, they were negatively linked. This inconsistency means that a scientist cannot simply pick the method that scores highest on faithfulness and expect it to be correct. The study explicitly rules out the idea that faithfulness is a reliable proxy for correctness, showing that a high score on one does not guarantee a high score on the other.
The audit also looked at how these explanations behave when the AI is confident versus when it is wrong. The researchers found that when the AI makes a confident but incorrect prediction, the explanation often becomes more faithful to the model's internal logic, even as it becomes less accurate regarding the chemistry. This creates a dangerous trap: the more confident the AI is in a wrong answer, the more convincing its explanation might appear, even though the explanation is leading the scientist astray. Furthermore, the study showed that the choice of the underlying AI architecture significantly impacts how well an explanation can locate the true chemical features, regardless of the explanation method used. This suggests that the reliability of an explanation is a property of the entire system, not just the tool used to generate the heat map.
Ultimately, this work serves as a necessary reality check for the field of artificial intelligence in chemistry. It demonstrates that the current practice of trusting AI explanations without verifying them against known chemical truths is risky. The researchers provide a framework for auditing these systems, but their primary conclusion is a negative one: there is no universal law that guarantees a faithful explanation is a correct one. For the medicinal chemist looking at a heat map, the study implies that the highlighted atoms are not necessarily the atoms that matter. The only way to know if an explanation is trustworthy is to test it against ground truth, a luxury that is rarely available in real-world drug discovery. Until that gap is closed, the most reliable approach is to treat these explanations as hypotheses to be tested, rather than facts to be accepted.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.