AI-Generated Responses to Patient's Messages: Effectiveness, Feasibility and Implementation

This study evaluating the implementation of Epic's GenAI tool for patient messaging in a Dutch academic hospital found that while initial adoption was high and drafts were well-structured, the tool ultimately failed to improve clinical efficiency or well-being, leading to declining usage due to barriers such as limited time savings, content inaccuracies, and misaligned expectations.

The Gist

Imagine you are a doctor or nurse at a busy hospital. At the end of a long day, instead of going home, you are buried under a mountain of digital letters from patients. Some ask about test results, some ask about medication, and some just want to know if they can come in for a check-up. This mountain of letters is heavy, stressful, and takes up time you could spend actually caring for people.

To help lift this mountain, the hospital tried a new tool: an AI "Ghostwriter."

Here is the story of how that experiment went, explained simply.

The Idea: A Helpful Co-Pilot

The hospital installed a special computer program (called "Art") inside their digital filing system. Think of it like a very smart, fast typist who has read every medical file in the building.

When a patient sends a message, the AI reads the patient's file and instantly writes a draft reply for the doctor.

• The Goal: The doctor just needs to read the draft, maybe tweak a sentence, and hit "send." This was supposed to save time and reduce stress, like having a co-pilot fly the plane while you just steer.

The Experiment: High Hopes, Mixed Results

The researchers watched how this worked for six months with doctors, nurses, and staff in four different departments (skin, lungs, cancer, and ears/nose/throat). They asked: Did it actually save time? Did people like it? Did it stop working after a while?

Here is what they found:

1. The "Honeymoon Phase" (The Beginning)

At first, everyone was excited. It was like getting a brand-new, high-tech gadget. People thought, "Wow, this will save me hours!"

• The Reality: The AI did write drafts quickly. It sounded polite and used the right medical terms.
• The Catch: The AI wasn't perfect. Sometimes it made up facts (hallucinations), sometimes it missed the point of the patient's question, and sometimes it sounded a bit robotic.

2. The "Reality Check" (The Decline)

As time went on, the excitement faded. It's like buying a fancy kitchen gadget that you use once, then realize it's actually more work to clean than just cooking by hand.

• Time Saved? Not really. The doctors found they still had to spend almost the same amount of time reading and fixing the AI's drafts as they would have just writing the letter themselves.
• Stress? It didn't go down. In fact, some doctors felt more stressed because they were worried, "Did the AI get the medicine dose right? If I miss a mistake, the patient could get hurt."
• Usage: At first, many people tried it. But after a few months, fewer and fewer people used it. It dropped from being a popular tool to something many ignored.

3. The "Trust Gap"

The biggest problem was trust.
Imagine the AI is a very eager intern. It writes a great letter, but it occasionally gets a patient's name wrong or suggests a medication they aren't actually taking.

• The doctors realized they couldn't just hit "send." They had to act like a strict editor, checking every single word.
• Because they had to check everything, the AI didn't feel like a "helper"; it felt like a "burden" that added a step to their workflow.

The Verdict: A Work in Progress

The study concluded that while the technology is impressive, it isn't quite ready to replace the human touch or save significant time yet.

• What worked: The AI was good at formatting, using a polite tone, and pulling basic info from the patient's file.
• What failed: It struggled with complex questions, medical accuracy, and fitting into the doctors' natural way of speaking.

The Lesson for the Future

The researchers say this isn't a failure of the technology, but a failure of expectations.

• The Analogy: You can't just hand a new driver a Ferrari and expect them to drive it perfectly on day one. They need training, clear rules, and a car that fits their driving style.
• The Fix: Hospitals need to teach doctors exactly how to use these tools, set clear rules about who is responsible for the final message, and keep improving the AI so it makes fewer mistakes.

In short: The AI "Ghostwriter" is a talented apprentice, but right now, the doctors still have to do most of the heavy lifting. It's a promising start, but we need to teach the robot how to be a better partner before it can truly lighten the load.

Technical Summary

1. Problem Statement

The increasing volume of patient-provider electronic messages (InBasket) has created a significant administrative burden for Healthcare Professionals (HCPs), contributing to burnout and exhaustion. While Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) offer a potential solution by drafting responses using Electronic Health Record (EHR) data, most existing evidence comes from US-based, English-language contexts. There is a critical lack of data regarding:

• The effectiveness and feasibility of GenAI tools in non-English healthcare settings (specifically Dutch).
• The long-term user experience, adoption rates, and perceived impact on clinical efficiency and well-being in real-world clinical workflows.
• Specific barriers and facilitators to implementing these tools across diverse clinical departments and professional roles.

2. Methodology

The study employed a hybrid type 1 effectiveness-implementation design at Radboud University Medical Center (Radboudumc) in the Netherlands.

• Intervention: Implementation of Epic's Augmented Response Technology (Art), a GenAI tool integrated into the EHR (Epic InBasket).
  • Workflow: Incoming patient messages (Type 54) are classified (administrative, general, medication, results) using GPT-3.5 Turbo. Contextual EHR data is retrieved based on the category, and GPT-4o (static model) generates a draft response.
  • User Interaction: HCPs can accept, modify, or discard the draft. They can also provide structured feedback (factual correctness, relevance, etc.).
• Study Population: 237 HCPs (physicians, nurses, support staff) across four departments: Dermatology, Medical Oncology, Otorhinolaryngology (ENT), and Pulmonology.
• Study Period: 6 months (January – June 2025).
• Data Collection:
  1. EHR Metrics: Efficiency indicators (review time, draft time) and adoption indicators (number of drafts generated, adoption rate, feedback counts).
  2. Surveys: Repeated measures at three time points:
     • PRE: Baseline (expectations).
     • POST-1: 1 month post-implementation.
     • POST-2: 4 months post-implementation.
     • Metrics: Well-being (NASA-TLX, PFI-WE), Clinical Efficiency (TAM-PU), Use/Feasibility (TAM-BI, TAM-ATT), and Usability (TAM-PEOU, SUS, NPS).
• Analysis:
  • Quantitative: Linear mixed models to analyze changes over time; descriptive statistics for EHR metrics.
  • Qualitative: Thematic analysis of open-ended survey responses using the NASSS framework (Nonadoption, Abandonment, Scale-up, Spread, and Sustainability).

3. Key Contributions

• Non-English Context Validation: Provides one of the first rigorous evaluations of an LLM-based drafting tool in a Dutch, non-English academic hospital setting.
• Longitudinal Implementation Data: Moves beyond initial pilot enthusiasm to capture the "reality check" of adoption and usability over a 6-month period, highlighting the decline in user satisfaction over time.
• NASSS Framework Application: Systematically categorizes barriers and facilitators across technology, adopter, and organizational domains, offering a structured approach for future GenAI implementation research.
• Workflow Integration Insights: Identifies specific friction points in the human-AI loop, such as the tension between time-saving expectations and the actual time required for verification.

4. Key Results

A. Usage and Adoption Metrics

• Volume: 8,410 drafts were generated in total.
• Adoption Rate: Only 16.7% of generated drafts were actually used by HCPs.
  • Highest adoption: Pulmonology (27.1%).
  • Lowest adoption: Medical Oncology (10.7%).
• Trend: Adoption was initially high but declined significantly over the 6-month period.
• Feedback: Only 2.2% of drafts were marked "useful," while 5.0% received constructive feedback. The most common feedback issues were factual incorrectness (37.5%) and lack of relevance (30.4%).

B. Effectiveness (Efficiency & Well-being)

• Time Savings: No significant difference was found in review time or draft composition time between messages handled with Art versus those without.
  • Review time: ~2m 2s (with Art) vs. ~2m 10s (without).
  • Draft time: ~1m 52s (with Art) vs. ~1m 56s (without).
• Perceived Efficiency: Perceived clinical efficiency and the usefulness of the tool declined significantly from PRE to POST-2.
• Well-being: No significant changes were observed in task load or work exhaustion scores.

C. Feasibility and Usability

• Usability Scores: Scores for ease-of-use and intention-to-use declined significantly over time.
• Net Promoter Score (NPS): Dropped from -13.04 (POST-1) to -37.93 (POST-2), indicating a shift from neutral to negative sentiment regarding recommending the tool.

D. Barriers and Facilitators (Qualitative)

• Barriers:
  • Technology: Medical inaccuracies (hallucinations), inability to handle complex/multi-part questions, and style misalignment (impersonal tone).
  • Adopter: Fear of automation bias (blindly trusting AI), need for constant verification (negating time savings), and lack of trust.
  • Workflow: Limited practical utility for specific correspondence types; the tool felt like an "extra step" rather than a shortcut.
• Facilitators:
  • Technology: Well-structured sentences, empathetic tone, and automatic retrieval of EHR data (e.g., lab values, gender).
  • Workflow: Potential for faster initial response times and reduced typing for simple queries.

5. Significance and Conclusions

• Expectation vs. Reality: The study reveals a stark contrast between high initial optimism and the reality of limited practical impact. The tool did not reduce the cognitive load or time burden as anticipated because the verification cost (checking for errors) outweighed the drafting savings.
• Implementation Lessons: Successful scaling requires more than just technical deployment. It necessitates:
  • Clear Communication: Managing expectations that the tool is an assistant, not an autonomous agent.
  • Workflow Integration: Establishing feedback loops for prompt engineering and tool refinement (e.g., integrating "Text Assistant" for style customization).
  • Governance: Defining clear quality indicators and thresholds for when a GenAI tool is robust enough for clinical use.
• Future Directions: The authors call for standardized evaluation frameworks, patient perspective inclusion, and iterative development cycles to address the "uncanny valley" of AI-generated medical text before widespread adoption.

Final Verdict: While GenAI tools like Art show promise for patient communication, the current implementation in this Dutch setting was not effective in reducing workload or improving well-being, and feasibility was low due to accuracy concerns and declining user trust. The study underscores the need for rigorous quality control and workflow adaptation before such tools can be responsibly scaled.