When Patients Say “Do Not Share”: LLMs Misroute Sensitive Information in Clinical Documentation
This study reveals that large language models frequently fail to honor patient requests to withhold sensitive information in clinical documentation, disclosing such data in over a third of cases to unrelated recipients, but demonstrates that implementing recipient-conditioned routing workflows can significantly mitigate this risk without compromising necessary information sharing.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet corners of a doctor's visit, patients often share things they would rather keep hidden. They might speak about a struggle with addiction, a private health condition, or a difficult home life, trusting that the clinician will use this information only to help them heal. Sometimes, a patient will explicitly ask that a specific detail not be shared with anyone else. This request is a fundamental part of medical trust, a promise that sensitive facts will stay within the circle of care. Today, doctors increasingly rely on artificial intelligence to listen to these conversations and write them down into official medical notes, referrals to other specialists, and letters for employers. These computer programs, known as large language models, are designed to understand human language and summarize it quickly. But a new question has emerged: when a patient says "do not share," does the machine listen? The answer matters because these digital summaries are not just records; they are documents that travel. A note meant for a heart doctor might accidentally end up in a letter sent to a boss, or a summary for a patient might be read by a family member who was not meant to see it. The safety of this system depends on the computer knowing not just what to say, but who should hear it.
Researchers at several major medical centers set out to test exactly this scenario. They wanted to see if current artificial intelligence systems could respect a patient's explicit request to withhold sensitive information when generating documents for different people. To do this, they created a massive, controlled experiment involving thousands of simulated medical encounters. They built 120 detailed case studies covering six sensitive areas of health, such as mental health, substance use, and reproductive issues. For each case, they created two versions: one where the patient simply shared the information, and another where the patient shared the same information but added a clear, direct request not to share it with anyone outside the immediate care team. They then asked seven different artificial intelligence models to turn these conversations into five different types of documents: a note for the treating doctor, a referral to a specialist who needed the information, a referral to a specialist who did not need it, a summary for the patient, and a letter for an employer.
The results revealed a significant gap between what patients ask for and what the machines deliver. Even when a patient explicitly asked that a sensitive fact be kept private, the artificial intelligence models still included that information in 34.2 percent of the documents sent to people who did not need to know, such as unrelated specialists or employers. In the tests using real-world language from patient-authored health questions, the problem was even more pronounced; without a request, the models shared the information in nearly 68 percent of the inappropriate documents, and even with the request, they still shared it in about 34 percent of cases. The models seemed to struggle with the logic of context. They treated the medical history as a single block of truth that belonged in every document, rather than understanding that a fact can be true and important for one doctor but completely inappropriate for another. The researchers found that simply telling the model "do not share" was not a reliable way to stop it from leaking sensitive details.
The study also uncovered a dangerous trade-off. When the researchers tried to fix the problem by giving the models stricter instructions to hide sensitive information, the machines sometimes went too far. They began to leave out facts that were actually necessary for safe medical care. For example, in cases where a patient asked to hide a condition that a receiving specialist absolutely needed to know about for safety, the models sometimes omitted the information entirely, potentially putting the patient at risk. This created a difficult situation where the system either shared too much or hid too much. The researchers found that the models did not have a built-in ability to weigh the patient's privacy request against the clinical need for the information. They simply followed the most recent instruction without understanding the consequences of their choices.
To address this, the team tested a more sophisticated approach called a recipient-conditioned routing workflow. Instead of just asking the model to write a note, this system first asked the model to identify the sensitive information, then decide if the specific recipient of that document needed to see it, and finally draft the text accordingly. When they tested this method on one of the models, it worked remarkably well, reducing the rate of inappropriate sharing from over 21 percent down to just 1.2 percent without causing the system to hide necessary medical facts. However, when they tried the same method on a different model, it caused a massive spike in harmful omissions, hiding critical information in nearly two-thirds of the cases. This showed that the solution is not universal; what works for one artificial intelligence system can break another. The study concludes that relying on a patient's verbal request alone is not enough to protect privacy in an automated world.
The findings suggest that before these tools are used widely in hospitals, they must be tested not just for how well they write, but for how well they understand who should read what. A document generated by a computer is not just a summary of facts; it is a decision about who gets to know a patient's story. If the system cannot distinguish between a note for a treating physician and a letter for an employer, it fails a basic safety test. The researchers emphasize that secure storage of data is not the same as secure sharing of information. Even if a system is locked down and the data is encrypted, the model itself can still place a sensitive fact in the wrong document. The study recommends that every type of document and every version of the software be checked separately for these specific errors. Until these systems can reliably honor a patient's request without compromising their safety, the final decision on what goes into a medical letter should remain in the hands of a human clinician.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.