🧬 biology

What Molecular Structure Cannot Tell Us: A Taxonomy of Explainability Gaps in GNN-Based Drug Toxicity Prediction

This study reveals that molecular structure alone explains only about 45% of acetylsalicylic acid's known adverse effects and introduces a four-category taxonomy to classify explainability gaps in GNN-based drug toxicity prediction, highlighting critical limitations for regulatory safety workflows.

Original authors: Juergen Dietrich

Published 2026-05-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Juergen Dietrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Question: Can a Picture of a Molecule Tell the Whole Story?

Imagine you are trying to predict how a specific car will behave in a race just by looking at a 2D blueprint of its engine. You know the blueprint perfectly. But can that blueprint tell you if the driver will get tired, if the tires will blow out on a specific track, or if the car will crash because of a sudden rainstorm?

This paper asks a similar question about drug safety. Scientists use powerful computer programs (called Graph Neural Networks or GNNs) to look at the "blueprint" of a drug molecule (its atomic structure) and predict if it will be toxic.

The author, Juergen Dietrich, wanted to know: How much of a drug's actual danger can be seen just by looking at its molecular blueprint?

To find out, he used Aspirin (Acetylsalicylic Acid) as a test case. Aspirin is like the "gold standard" of drugs; we know everything about it because people have taken it for 125 years.

The Experiment: The "Blind" Computer

The computer was trained on a massive dataset called Tox21, which contains results from 12 different lab tests (assays) designed to spot toxicity. Think of these 12 tests as 12 different "safety inspectors" looking for specific types of problems (like "does this burn the skin?" or "does this damage DNA?").

The computer looked at the Aspirin blueprint and tried to predict if it would fail any of these 12 tests.

The Result: The computer said, "No, Aspirin looks safe."
The Reality: We know Aspirin does cause side effects (like stomach bleeding and ringing in the ears).

The computer was wrong. But why? The paper creates a four-category "Gap Taxonomy" to explain exactly where the computer failed.

The Four "Gaps" (Why the Computer Missed the Danger)

The author categorizes the reasons the computer failed into four types of "blind spots":

1. GAP-1: The "Missing Context" Gap (The Driver's Fatigue)

What it is: Some dangers depend on the person taking the drug, not the drug itself.
The Analogy: A blueprint can't tell you if the driver is 80 years old, has a specific genetic condition, or is taking another medicine that conflicts with the car.
In the Paper: About 36% of Aspirin's side effects (like Reye's syndrome in children or allergic reactions) fall here. The molecular structure simply doesn't contain this information. No amount of better computer code can fix this.

2. GAP-2: The "Missing Data" Gap (The Secret File)

What it is: The danger is in the structure, but the computer has never seen the data because it's hidden in private company files.
The Analogy: Imagine the blueprint is perfect, but the safety manual for this specific car model is locked in a vault at the factory. The computer only has access to public safety manuals, which don't mention this specific risk.
In the Paper: The author searched public databases for data on a specific mechanism (mitochondrial uncoupling) and found 42 documented tests but zero public results. The data exists, but it's "Missing Not At Random" (MNAR)—it's missing because companies keep it private. This accounts for about 9% of the gaps.

3. GAP-3: The "Wrong Tool" Gap (The Wrong Inspector)

What it is: The computer is trained on the wrong set of tests.
The Analogy: You hired 12 safety inspectors, but none of them are trained to check for "stomach bleeding." They are all experts in "fire safety" and "brake failure." Even if the car has a stomach-bleeding problem, these inspectors will say, "Everything looks fine," because that's not what they are looking for.
In the Paper: 50% of the failures happened here. The Tox21 tests (the 12 inspectors) don't measure the specific biological mechanism that causes Aspirin to hurt the stomach (COX-1 inhibition). The computer can't predict a problem it was never taught to look for.

4. GAP-4: The "Wrong Focus" Gap (The Distracted Student)

What it is: The computer sees the right parts of the molecule but focuses on the wrong ones.
The Analogy: Imagine a student taking a test about why a car engine overheats. The student looks at the engine and says, "It's overheating because of the color of the paint!" The student is looking at the car, but they are focusing on the wrong feature.
In the Paper: For a specific toxicity (mitochondrial damage), the computer focused on the ring shape of the Aspirin molecule (like a benzene ring) instead of the acid group (the carboxyl group) that actually causes the problem.
- The author tested this by changing how the computer "pays attention" to atoms. Even when the computer was forced to pay attention to the acid group, it still failed.
- The Discovery: The mistake happened deep inside the computer's "thinking layers" (message passing), not just at the final step. The computer had learned a bad habit from its training data (associating rings with toxicity) that it couldn't unlearn just by changing the final calculation. This accounts for about 9% of the gaps.

The Bottom Line

The study concludes that for Aspirin, looking at the molecular structure alone explains only 45% of its known side effects.

45% is "MATCH": The structure holds the answer, and the computer could find it if the data and tools were perfect.
55% is "GAP": The answer is missing because of patient context (36%), hidden private data (9%), or the wrong tests being used (9%).

Why This Matters (According to the Paper)

The author argues that we need to stop assuming that if a computer says "safe," the drug is safe.

If a drug class (like NSAIDs/Aspirin) has a 50% mismatch with the available safety tests, we are flying blind for half the risks.
We cannot just build "smarter" computers (better AI models) to fix this. We need better data (getting access to private company files), better tests (creating new lab tests that actually measure stomach bleeding), and we must accept that some risks (like patient allergies) will never be predictable from a molecule's shape alone.

The paper provides a "map" (the Taxonomy) to help regulators and scientists understand exactly why a prediction failed, rather than just saying the AI is "wrong."

Technical Summary: A Taxonomy of Explainability Gaps in GNN-Based Drug Toxicity Prediction

Problem Statement

While Graph Neural Networks (GNNs) offer a structurally natural approach to molecular toxicity prediction by operating directly on atomic connectivity, a fundamental question remains underexplored: what fraction of a drug's known pharmacological profile is actually encodable in its molecular structure? As regulatory frameworks like Good Pharmacovigilance Practice (GVP) and New Approach Methodologies (NAMs) increasingly integrate AI, there is a critical need to understand the limits of structure-based prediction. Specifically, it is unclear whether prediction failures stem from inherent biological complexity, data scarcity, assay mismatches, or model representation errors.

Methodology

The study employs Acetylsalicylic Acid (ASA/Aspirin) as a model compound due to its extensive clinical history, well-characterized mechanisms (both on-target and off-target), and small size (13 heavy atoms) which facilitates atom-level interpretability.

Model Architecture and Training

Model: A Message Passing Neural Network (MPNN) based on Neural Network Convolution (NNConv) [Gilmer et al., 2017].
Input: Molecular graphs constructed via RDKit with 31-dimensional atom features and 12-dimensional bond features.
Training: Multi-task learning on the Tox21 benchmark (12 toxicological assay endpoints). The model uses a global mean pooling layer and a two-layer MLP classifier.
Explainability: GNNExplainer [Ying et al., 2019] is applied to generate atom-level attribution masks. A "Pharmacophore Ratio" metric is defined to quantify whether the model focuses on the correct chemical substructure (e.g., the carboxyl group vs. the aromatic ring).
Control Experiment: An Attention Pooling mechanism [Li et al., 2015] is implemented to test if representation errors originate in the aggregation step or the message-passing layers.

Gap Taxonomy Construction

The authors define a four-category taxonomy to classify prediction failures:

GAP-1 (Non-encodable): Effects requiring patient-level context (age, genetics) absent from the molecular graph.
GAP-2 (Data Gap / MNAR): Effects structurally predictable in principle but missing from public training data due to "Missing Not At Random" (MNAR) mechanisms (e.g., proprietary data).
GAP-3 (Assay Panel Mismatch): The training panel (Tox21) lacks assays measuring the relevant biological mechanism.
GAP-4 (Representation Error): The model learns the wrong structural feature as a predictor due to confounding in the training data, despite the correct feature being present.

Empirical Validation

MNAR Quantification: A systematic query of the ChEMBL database was performed for SR-MMP (Mitochondrial Membrane Potential) active carboxylic acids to quantify data gaps.
Mapping: A manual mapping was created between Tox21 endpoints and clinical adverse effects documented in DrugBank and SIDER.

Key Results

Model Performance and Prediction

The MPNN achieved a mean ROC-AUC of 0.69 across Tox21 endpoints, with Stress Response (SR-) endpoints performing better than Nuclear Receptor (NR-) endpoints.
Critical Finding: When evaluating ASA, the model classified all 12 Tox21 endpoints as negative (below the 0.5 threshold). This includes SR-MMP (AUC 0.95 in training) and SR-ARE, which are pharmacologically relevant to ASA toxicity. The model failed to predict known adverse effects despite high training performance on the general dataset.

Attribution Analysis (GAP-4)

For the SR-MMP endpoint, GNNExplainer revealed a Pharmacophore Ratio of 0.646 under mean pooling. This indicates the model incorrectly focused on the aromatic ring (atoms 4–9) rather than the pharmacologically relevant carboxyl group (atoms 10–12), which drives the proton carrier mechanism.
Attention Pooling Experiment: While the attention gate correctly upweighted carboxyl atoms ( $\alpha = 0.62$ ), the Pharmacophore Ratio only improved to 0.805, failing to reach the "correct focus" threshold. This confirms that the representation error resides in the message passing layers (where embeddings become ring-dominated due to training data confounding with polycyclic aromatic hydrocarbons), not in the aggregation step.

Gap Taxonomy Quantification

Clinical Adverse Effects: Of 11 known ASA adverse effects, only 45% (5/11) are structurally encodable (MATCH).
- GAP-1 (Non-encodable): 36% (e.g., Reye's syndrome, poisoning).
- GAP-2 (MNAR): 9% (e.g., Tinnitus; 42 documented assays found 0 retrievable bioactivity entries for carboxylic acids in ChEMBL).
- GAP-3 (Assay Mismatch): 18% (e.g., GI bleeding, platelet inhibition).
- GAP-4 (Representation Error): 9% (e.g., Hepatotoxicity).
Tox21 Panel Coverage: 50% (6/12) of Tox21 endpoints represent a GAP-3 mismatch for NSAIDs, as the panel does not measure COX-1 inhibition or platelet dysfunction mechanisms.

Significance and Claims

The paper argues that the limitations of GNN-based toxicity prediction are not merely a matter of model architecture or data volume, but are structural and systematic.

Taxonomy of Failure: The study introduces a four-category Gap Taxonomy (GAP-1 to GAP-4) that distinguishes between fundamentally non-encodable effects, data gaps (MNAR), assay mismatches, and representation errors. This framework moves beyond aggregate metrics like AUC to diagnose specific root causes of failure.
MNAR Empirical Evidence: The study provides empirical evidence of the MNAR problem in public databases, showing that critical data for specific mechanisms (e.g., mitochondrial uncoupling by lipophilic acids) is absent from public repositories, residing instead in proprietary DMPK databases.
Localization of Representation Errors: The attention pooling experiment demonstrates that representation errors (GAP-4) can originate in the message passing layers of MPNNs due to confounding training distributions, and that simple architectural changes (like attention pooling) may not suffice to correct them without addressing the underlying data bias.
Regulatory Implications: The findings suggest that current High-Throughput Screening (HTS) panels (like Tox21) and NAMs may have structural blind spots for specific drug classes (e.g., NSAIDs). The authors claim that a 50% panel mismatch for NSAIDs constitutes a quantifiable limitation in nonclinical safety coverage, implying that mechanistic absence in HTS data should not be interpreted as a negative safety signal without supplementary evidence.

The study concludes that resolving these gaps requires targeted interventions: COX-specific assay panels (for GAP-3), federated learning for proprietary data access (for GAP-2), and integration of post-marketing pharmacovigilance databases (for GAP-1 and GAP-2).