What Can Artificial Intelligence Learn from Medicine? Generative Analogies and Reliable Machine Learning Systems

In the last few years, doctors and computer scientists have begun working together to build artificial intelligence tools that can help diagnose diseases, predict patient outcomes, and suggest treatments. These tools, known as machine learning systems, are powerful because they can find patterns in massive amounts of data that humans might miss. However, there is a nagging problem: these systems often work like black boxes. They produce results without explaining how they reached them, and they rely on statistical connections rather than clear theories about why a treatment works. This creates a dilemma for doctors who need to trust the tools they use. If a computer says a patient has a disease, but no one understands why it said that, how can a doctor be sure it is right? This uncertainty has led to a difficult question: how do we verify that these complex, opaque systems are reliable enough to be used in real hospitals?

Two philosophers, Emanuele Ratti and Lena Zuchowski, have proposed a new way to think about this problem by looking at how medicine itself handles similar uncertainties. They suggest that we should not try to force artificial intelligence to become transparent or explainable in the way we usually expect. Instead, they argue that we should learn from the long history of how medicine validates new drugs. Just as doctors do not always know exactly how a specific pill works inside the body, they still know it is safe and effective because of the rigorous process used to test it. The researchers propose that the same kind of rigorous testing process used for drugs can be adapted to create a new standard for trusting artificial intelligence.

The core of their argument rests on a comparison between two very different processes: the development of new medicines and the building of machine learning systems. In medicine, when a new drug is discovered, scientists often do not fully understand the biological mechanism behind it. They might not know exactly how the molecule interacts with the body's cells. Despite this lack of theoretical knowledge, the drug can still be proven to work. This is achieved through a process called clinical translation, where researchers systematically test the drug under many different conditions. They figure out the right dosage, the specific types of patients it helps, and the situations where it might fail. By gathering this information, they create a detailed profile of the drug's usefulness, even if the underlying "why" remains a mystery.

Ratti and Zuchowski argue that machine learning systems face the exact same situation. These systems are often "atheoretic," meaning they do not rely on human-made theories; they are "associationist," meaning they find links between data points without understanding the cause; and they are "opaque," meaning their internal decision-making is hidden. Because of this, trying to make the AI explain itself is often impossible. Instead of fighting this nature, the authors suggest we should copy the medical approach. We should stop trying to demand that the AI explain its logic and start demanding that we know the specific conditions under which it works.

To make this idea concrete, the researchers developed a new framework they call a "learning ensemble." Think of this as a detailed instruction manual that goes far beyond just saying "this computer program works." In medicine, a drug is not just a chemical; it is a chemical used in a specific way, for a specific group of people, with specific side effects. Similarly, a machine learning system is not just a piece of code; it is a system that works only when certain conditions are met. The learning ensemble is a collection of information that defines those conditions. It includes details about the data used to train the system, the specific hardware it runs on, the types of patients it was tested on, and the exact metrics used to judge its success.

The authors break this information down into three main categories. The first is the "boundaries of reliability." This section documents the environment in which the system was built. It asks questions like: Was the data clean and diverse, or was it messy and limited? Was the system tested on powerful computers, or did it struggle with basic hardware? If the data came from only one hospital, the system might fail when used in a different hospital with different equipment. By documenting these boundaries, users can know exactly where the system is safe to use and where it might break.

The second category is "performance." This is not just a single number saying the system is 90% accurate. Instead, it requires a detailed report of how the system performs in different situations. For example, does it work equally well for young and old patients, or for people of different genders? Does it make more mistakes when the data is slightly different from what it was trained on? The researchers emphasize that knowing where a system fails is just as important as knowing where it succeeds. If a system is great at diagnosing one type of cancer but terrible at another, that specific failure must be recorded so doctors do not use it for the wrong cases.

The third category is the "functional dimension." This looks at what the system is actually supposed to do in the real world. A system might be mathematically perfect at predicting a number, but if that number does not help a doctor make a better decision, the system is useless. This part of the framework checks if the system's output matches the needs of the medical team. For instance, if a system is designed to help a doctor decide whether to admit a patient to the hospital, it must be tested to see if it actually helps with that specific decision. If the system is used for a purpose it was not designed for, it becomes unreliable, regardless of how well it performs its original task.

The researchers illustrate the power of this approach with real-world examples of failure. They point to a study where an artificial intelligence system claimed to detect cancer by looking at the genetic makeup of bacteria in the human body. The system seemed to work perfectly at first, with very high accuracy. However, when the researchers looked closely at the data, they found that the system had not actually learned about cancer at all. It had learned to recognize a specific type of error in the data collection process that happened to be more common in cancer patients. Because the system was opaque, this mistake was hidden. If the researchers had used the "learning ensemble" framework, they would have been required to document the data sources and the specific conditions of the test. This documentation would have revealed that the data was flawed and that the system was relying on a shortcut rather than a real biological connection.

Another example involves a system designed to predict which patients with pneumonia were at high risk of dying. The system learned that patients with asthma were actually at lower risk. This seemed counterintuitive. The reason was that the system had learned that asthmatic patients received very aggressive and effective treatment in the hospital, which lowered their risk of death. However, if a doctor used this system to decide who to treat, they might mistakenly think asthmatic patients did not need care. The system was mathematically correct based on the data, but it was functionally dangerous because it did not understand the context of medical treatment. The learning ensemble framework would have caught this by requiring a clear definition of the system's purpose and a check against real-world medical knowledge.

The authors are careful to note that this framework does not solve the mystery of how machine learning works. The systems will still be opaque, and they will still rely on statistical patterns rather than clear theories. The value of the learning ensemble is not in making the AI transparent, but in making the use of the AI transparent. It shifts the focus from asking "How does the computer think?" to asking "Under what exact conditions can we trust the computer's answer?" This is a subtle but crucial shift. It allows doctors to use powerful tools without needing to understand the complex mathematics behind them, as long as they have a clear map of where those tools are safe to use.

This approach suggests a new way to regulate and evaluate artificial intelligence in medicine. Instead of demanding that every algorithm be explainable, regulators and hospitals should demand a complete "learning ensemble" for every system. This document would act as a certificate of reliability, detailing the data, the performance, and the intended use. If a hospital wants to use a system, they can check the ensemble to see if their specific situation matches the conditions where the system has been proven to work. If the conditions do not match, the system should not be used. This method accepts the limitations of current technology while providing a robust way to manage the risks.

The paper concludes that this framework is a practical step forward. It does not claim to have solved all the problems of artificial intelligence, nor does it promise that these systems will never make mistakes. Instead, it offers a structured way to build trust. By treating machine learning systems like medical treatments—where the focus is on the conditions of use rather than the internal mechanism—we can create a safer, more reliable path for integrating these technologies into healthcare. The goal is not to make the black box see-through, but to build a reliable guide that tells us exactly where it is safe to look inside.