When Epistemic Delegation Becomes Abdication: The Limits of Clinical Authority in AI-Mediated Medicine
This paper argues that while clinical medicine has always relied on external expertise, AI-mediated question-generation creates a distinct risk of epistemic abdication unless clinicians retain meaningful control through intelligibility, contestability, and situated verification at the point of care.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
For centuries, doctors have relied on tools they cannot build themselves and knowledge they cannot fully reconstruct from scratch. A cardiologist trusts the echo of a heart sound captured by a machine she did not design; a surgeon relies on a pathology report written by a specialist she has never met. This dependence is not a flaw in medicine but its very foundation. Complex care requires a network of trust, where one person's expertise stands in for another's. In recent years, artificial intelligence has joined this network, offering to measure vital signs, flag potential risks, and even suggest diagnoses. But as these systems grow more capable, a quiet question has emerged: at what point does trusting a machine stop being a sensible division of labor and start becoming a surrender of judgment?
The core of this issue lies in how we define the relationship between a doctor and the technology they use. When a doctor uses a stethoscope, the tool amplifies a sound, but the doctor decides what that sound means. When a doctor uses a blood test, the machine provides a number, but the doctor interprets it in the context of the patient's story. This paper argues that artificial intelligence is beginning to change the nature of this relationship in a specific and dangerous way. It suggests that while doctors have always depended on tools, they are now facing a new risk where the tool itself begins to decide what questions are worth asking in the first place. The author, Andrea Mangiameli, proposes that this shift moves medicine from a state of "delegation," where a doctor remains in charge, to "abdication," where the doctor effectively hands over the authority to know what is happening to a system they cannot truly understand or check.
The paper distinguishes between three ways doctors currently use outside help. The first is measurement, where a machine simply counts or records a value, like a thermometer measuring temperature. The second is selection, where a system filters through vast amounts of data to highlight only the most important bits, like a security camera that only alerts a guard when it sees a specific type of movement. The third, and most critical, is question generation. This is where the system does not just answer a known question or highlight a known risk, but suggests entirely new patterns or categories of illness that doctors had not previously considered. For example, a system might analyze a patient's continuous movement and sleep data to suggest a new kind of physiological signature that predicts heart failure, even though no doctor had ever named or defined that pattern before.
The danger arises when this third function—suggesting new questions—becomes the primary way doctors make decisions, especially if the system is opaque. Opacity here does not mean the software is a secret; it means that even if a doctor sees how the system works, the sheer complexity of the data and the math makes it impossible for a human mind to hold the logic in their head. The system might be right, and it might be statistically validated in large studies, but the doctor at the bedside cannot explain why the system is making a specific claim for a specific patient, nor can they easily check that claim with a different method. The paper argues that when a system carries enough weight to change a patient's treatment plan, but the doctor cannot understand the reasoning, contest the conclusion, or verify it with an independent check in that moment, the doctor has lost control. This is not a failure of the doctor's skill, but a failure of the environment in which they work.
The author introduces a framework called "meaningful human control" to describe the conditions needed to prevent this loss of control. For a doctor to remain in charge, three things must be true. First, the system must be intelligible: the doctor must be able to understand, in general terms, what the system is looking for and what kind of mistakes it might make. Second, there must be contestability: there must be a real, practical way for the doctor to question the system's output, pause its advice, and weigh it against other evidence, including the patient's own description of how they feel. Third, and perhaps most importantly, there must be situated verifiability. This means that at the specific moment of care, the doctor must have access to an independent way to check the system's claim. This could be a second test, a different kind of observation, or a comparison with the patient's history. If a system suggests a new diagnosis based on a complex pattern, but the only way to check that diagnosis is to run the same complex system again, then the doctor has no independent leverage. They are forced to trust the machine blindly.
The paper clarifies that this is not an argument against using artificial intelligence. Medicine has always relied on instruments and experts that individual doctors cannot fully replicate. The argument is specifically about the conditions under which that reliance remains safe. The author suggests that the risk is highest when a system proposes a new clinical category—a new way of seeing a patient's body—before that category has been tested, debated, and agreed upon by the medical community. In the past, new medical discoveries moved slowly, allowing time for doctors to argue over them and establish how to verify them. Artificial intelligence can move much faster, surfacing patterns instantly. If the medical system does not build the infrastructure to verify these new patterns immediately, doctors may find themselves acting on suggestions they cannot validate.
Ultimately, the paper concludes that the problem is not the technology itself, but the clinical environment that surrounds it. A system can be highly advanced and still be governed responsibly if the hospital or clinic ensures that doctors have the training, the procedures, and the alternative tools needed to check the system's work. Conversely, even a simple system can become dangerous if the environment forces doctors to rely on it without the ability to question or verify its advice. The author warns that when practical authority—the power to direct a patient's care—outpaces the ability to understand and check that authority, the relationship shifts from delegation to abdication. In this state, the doctor is no longer the one answering for the patient's care; the system is, and the doctor is left without the tools to answer for it. The solution, the paper argues, is to ensure that as medicine becomes more automated, the conditions for human oversight are built into the system with the same care as the technology itself.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.