Interpretability Methods for Causal Predictions in Biomedical AI: A Systematic Review of Approaches and Evaluation Practices
This systematic review of 47 biomedical AI studies reveals that while interpretability methods like SHAP and LIME are widely used to explain causal prediction models, their evaluation overwhelmingly relies on property-based metrics and rarely validates actual causal validity, highlighting a critical gap between model explanation and causal inference.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of modern medicine, artificial intelligence has become a powerful tool for spotting patterns that human eyes might miss. These computer systems can scan thousands of medical images or sift through years of patient records to predict who might get sick or how a disease will progress. However, there is a crucial difference between knowing that two things happen together and knowing that one thing actually causes the other. A computer might learn that a specific skin mark often appears alongside a serious illness, but that does not mean the mark causes the sickness; it might just be a side effect of something else entirely. If a doctor relies on a computer that only sees these connections, they might make the wrong decision when trying to treat a patient, because treating the mark will not cure the disease. This gap between spotting a pattern and understanding the true cause is the central challenge that researchers are trying to solve.
A new systematic review by Albert Schulle from the Technical University of Munich investigates how well current tools help us understand these causes. The study looked at hundreds of papers to see if the methods used to explain how AI makes its predictions are actually capable of revealing true biological causes. The researchers found that while scientists are eager to use these explanation tools, they are mostly using the wrong ones for the job. The most popular tools, which assign importance scores to different features like age or blood pressure, are excellent at showing what the computer noticed, but they are terrible at showing what actually caused the outcome. In fact, the review discovered that explicit assessment of causal validity was rare, appearing in only 9% of the studies examined, meaning the vast majority never actually tested whether their explanations were telling the truth about cause and effect.
The review examined 47 studies published between 2016 and 2025 that tried to apply these explanation methods to medical problems where cause and effect matter, such as predicting how a patient will respond to a specific drug. The researchers found that the vast majority of these studies, about 68 percent, relied on feature attribution methods. These are techniques that highlight which input variables the computer model used most heavily to make a decision. The two most common tools were SHAP and LIME, which act like a spotlight, shining on the data points the model found most important. While these tools are very good at showing which numbers the computer looked at, the review argues that they are fundamentally limited because they only describe associations. They can tell a doctor that high blood pressure was a key factor in a risk prediction, but they cannot tell the doctor if lowering the blood pressure will actually prevent the heart attack.
The study also looked for other, more advanced methods that might get closer to the truth. About 15 percent of the studies used counterfactual explanations, which imagine what would have happened if a patient had done something differently, such as taking a different medication. These are conceptually closer to how doctors think about cause and effect, but they were still used in only a small fraction of the research. Even more rare were concept-based methods, which try to explain decisions using human-understandable ideas like "tumor size" rather than raw data points, appearing in only 6 percent of the studies. The review found that the field is heavily skewed toward the simpler tools that explain associations, creating a risk that doctors will mistake a correlation for a cause.
Perhaps the most striking finding concerns how these explanations are tested. When researchers build a model and explain it, they need to check if the explanation is any good. The review found that 72 percent of the studies checked for fidelity, which simply means checking if the explanation matches what the computer model actually did. Another 43 percent checked for stability, ensuring the explanation didn't change randomly every time the model ran. These are important technical checks, but they do not answer the most important question: is the explanation right about the cause? The review found that only 9 percent of the studies ever tried to validate whether the explanation actually reflected a true causal relationship. In the few cases where researchers did try to check for causality, they mostly relied on asking human experts if the explanation "felt" plausible, rather than using rigorous tests to prove it.
This gap between what the tools do and what the field needs is significant. The author points out that using a tool designed to explain associations to solve a causal problem is like using a map of traffic patterns to plan a hiking route; the map might show where cars go, but it won't tell you which path is safe for walking. The review suggests that the current approach is methodologically fragmented, with researchers applying powerful explanation tools to models that were never built to understand cause and effect in the first place. The paper concludes that for artificial intelligence to be truly trustworthy in medicine, the field must move beyond just checking if an explanation looks consistent. It needs to develop new ways to test if an explanation is actually correct about the underlying biology, using better benchmarks and clearer assumptions about how diseases work. Until then, the promise of AI that can truly explain why a patient is sick remains out of reach.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.