💬 NLP

The Daily Dose: Workflow-Integrated Large Language Model Automation for Clinical Summarization and Trial Identification in Radiation Oncology

This study presents the design and early evaluation of "The Daily Dose," an LLM-driven system integrated into radiation oncology workflows that automatically generates patient summaries and identifies clinical trials, demonstrating high clinician satisfaction, excellent reliability, and significant perceived time savings in a mixed-methods survey of 55 respondents.

Original authors: Jason Holmes, Federico Mastroleo, Mariana Borras-Osorio, Srinivas Seetamsetty, Satomi Shiraishi, Mirek Fatyga, Judy C. Boughey, Cornelius A. Thiels, William G. Breen, Daniel J. Ma, Daniel K. Ebner, Da

Published 2026-05-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Holmes, Federico Mastroleo, Mariana Borras-Osorio, Srinivas Seetamsetty, Satomi Shiraishi, Mirek Fatyga, Judy C. Boughey, Cornelius A. Thiels, William G. Breen, Daniel J. Ma, Daniel K. Ebner, David M. Routman, Brady S. Laughlin, Carlos E. Vargas, Samir H. Patel, Sujay A. Vora, Nadia N. Laack, Andrew Y. K. Foong, Wei Liu, Mark R. Waddle

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a radiation oncologist's morning as a pilot preparing for a complex flight. Before takeoff, the pilot needs to check hundreds of data points: weather reports, fuel levels, engine status, and passenger manifests. In a hospital, this "pre-flight check" involves reading through massive electronic health records (EHRs) for dozens of patients. It's a mountain of paperwork that can feel overwhelming, leading to fatigue and less time for the actual "flight"—caring for the patient.

This paper introduces a new tool called "The Daily Dose" (TDD), which acts like an intelligent co-pilot for these doctors. It is a system that uses advanced AI (specifically a Large Language Model, or LLM) to do the heavy lifting of reading and summarizing patient files before the doctor even arrives at the hospital.

Here is a breakdown of how it works, what the study found, and what the authors concluded, using simple analogies.

The Problem: The "Data Deluge"

Radiation oncology generates a huge amount of data for every patient over time. Doctors have to jump between different software programs to piece together a patient's history, current treatment, and test results. Doing this manually is slow and mentally exhausting. The authors describe this as a major source of inefficiency and "cognitive fatigue" (brain tiredness).

The Solution: The "Intelligent Co-Pilot"

The team at Mayo Clinic built The Daily Dose. Think of it as a smart assistant that wakes up at 5:00 AM every day and does the following:

Checks the Schedule: It looks at the doctor's calendar to see who they are seeing that day.
Reads the Files: It dives into the hospital's digital records for each patient. It pulls out diagnosis details, treatment plans, lab results, and past notes.
Writes a Summary: Instead of making the doctor read 50 pages of notes, the AI writes a short, clear paragraph for each patient. It highlights the most important things, like "Patient X is on their 10th treatment day," or "Patient Y's PSA levels have dropped."
- Special Feature: If the patient has prostate cancer, the AI automatically calculates specific risk scores (like a "traffic light" system for cancer risk) based on the data.
Finds Clinical Trials: For new patients, the AI acts like a detective. It compares the patient's condition against a database of active research studies (clinical trials) at Mayo Clinic. If a patient might qualify for a study, the AI lists the trial and explains why they might fit.
Delivers the Report: At the start of the day, the doctor receives a single email with all these summaries. It's like getting a personalized briefing packet before leaving the house.

The Experiment: Testing the Co-Pilot

The researchers tested this system across three Mayo Clinic locations (Arizona, Florida, and Minnesota). They didn't just look at the code; they asked the people using it, "How does this feel?"

Who was asked?
They surveyed 55 healthcare workers, mostly radiation oncologists (the pilots), but also nurses and other providers.

What did they find?
The results were a mix of "Great!" and "It needs work," but mostly positive.

Adoption: Most people (about 84%) used the tool daily or several times a week. It stuck.
Satisfaction: On a scale of 1 to 5, the average satisfaction was high (around 3.9). Most users found it easy to use and well-organized.
Time Savings: This is where it got interesting.
- Some doctors felt they saved 20 minutes or more a day.
- Others felt they saved 5 to 10 minutes.
- About 22% said they didn't save any time at all.
- The Key Insight: The people who felt they saved time were the happiest. If the tool didn't save time, they weren't as happy with it.
Usefulness: About 60% felt it helped them stay updated on appointments and understand patient summaries better. However, opinions were split on whether it actually helped with team coordination or reduced the time spent reviewing charts. Some felt it was just a "nice-to-have" summary, not a time-saver.
The "Trial" Feature: Doctors liked the idea of the AI suggesting clinical trials, but some noted that the AI sometimes suggested trials that didn't quite fit the patient's criteria. It was a helpful starting point, but not perfect.

The "Glitches" in the System

The paper is honest about the tool's limitations. In the open-ended feedback, doctors mentioned:

Missing Info: Sometimes the AI couldn't find enough data to write a summary, so it had to say, "Not enough info."
Inaccuracies: Occasionally, the summary included unnecessary details or missed critical information.
Trial Errors: Sometimes the AI suggested clinical trials that the patient actually didn't qualify for, requiring the doctor to double-check.
External Records: The system currently only looks at records inside Mayo Clinic. If a patient has records from a different hospital, the AI can't see them, which limits its ability to give a complete picture.

The Conclusion: A Promising Start

The authors conclude that The Daily Dose is a successful first step. It proved that an AI tool can be embedded into a real, busy hospital workflow and that doctors are willing to use it.

It works: It is usable and generally valued.
It helps: It improves "situational awareness" (knowing what's going on with a patient before walking into the room).
It needs tuning: It's not perfect yet. The AI sometimes makes mistakes or misses data.

What's Next?
Based on this study, the team has already built The Daily Dose 2.0. This new version is smarter:

It can now pull in records from outside hospitals (fixing the "missing info" problem).
It uses a two-step process to save money on computing costs.
It has a completely redesigned system for matching patients to clinical trials to make it more accurate.
They plan to roll it out to every doctor in the Mayo Clinic Comprehensive Cancer Center.

Summary in One Sentence

The paper describes a new AI tool that acts as a morning briefing assistant for cancer doctors, summarizing patient files and suggesting research studies; while users found it helpful and easy to use, they noted it still needs refinement to be perfectly accurate and truly save time for everyone.

Technical Summary: The Daily Dose – Workflow-Integrated LLM Automation for Radiation Oncology

Problem Statement
Radiation oncology workflows generate vast amounts of heterogeneous data within Electronic Health Records (EHRs), creating a significant cognitive burden for clinicians who must synthesize this information daily. The manual process of accessing multiple software programs to gather patient history, treatment plans, and diagnostic reports contributes to inefficiency and reduced time for direct patient interaction. While Large Language Models (LLMs) have shown potential for automating information synthesis, there is a lack of evidence regarding real-world, workflow-embedded implementations at scale. Existing solutions often remain experimental or lack integration into the daily operational rhythm of clinical care.

Methodology
The authors developed and evaluated "The Daily Dose" (TDD), a production-level application integrated directly into the clinical workflow of Mayo Clinic's Radiation Oncology departments across three campuses (Arizona, Florida, and Minnesota).

System Architecture: TDD operates as a fully automated, provider-facing application running within a HIPAA-compliant secure computing environment. It utilizes RadOnc-GPT, a specialized LLM agent based on GPT-4o (temperature set to 0), as its reasoning engine.
Workflow Execution: Each morning at 5:00 AM local time, a coordinating script identifies the daily schedule for participating physicians. For every scheduled appointment, the system retrieves relevant EHR elements (diagnosis, treatment plans, radiology/pathology reports, clinical notes) and prompts RadOnc-GPT to generate content.
Core Functions:
1. Clinical Summarization: The system synthesizes retrieved data into concise, context-aware narratives describing the patient's current condition. It includes disease-specific parameters (e.g., PSA trends, Gleason scores, NCCN risk categories) when relevant. If data is insufficient, it outputs a standardized fallback statement.
2. Clinical Trial Identification: For new or consult visits, the system evaluates eligibility for clinical trials. It generates condition-intervention keyword pairs, iteratively searches the clinicaltrials.gov API (restricted to open, accruing trials at Mayo Clinic), and assesses inclusion/exclusion criteria against patient data.
3. Delivery: Summaries and trial matches are compiled into a single, physician-specific email delivered before the clinical day begins. The email includes appointment details, visual indicators (emojis), clinical summaries, and trial eligibility results.
Evaluation Design: A mixed-methods evaluation was conducted one month post-deployment. A structured, anonymous questionnaire was administered to all health workers receiving TDD. The survey, adapted from the System Usability Scale (SUS) and Technology Acceptance Model (TAM), assessed demographics, usage patterns, usability, satisfaction, perceived usefulness, and impact on workflow. Statistical analysis included descriptive statistics, Cronbach's alpha for reliability, and non-parametric tests (Spearman, Kruskal-Wallis, Mann-Whitney U) for correlations.

Key Results
The study received responses from 55 participants (50% response rate), the majority of whom were attending physicians (69.1%) in radiation oncology (94.5%).

Adoption and Usage: 83.6% of respondents reported using TDD daily or several times per week.
Usability and Satisfaction: The system received a mean usability and satisfaction score of 3.89/5. 87% agreed the tool was easy to use, and 76% found the summaries clearly structured.
Perceived Usefulness: The mean score for perceived usefulness was 3.43/5. 62% agreed it helped keep them updated on appointments, and 53% felt it improved understanding of patient summaries.
Time Savings: Perceived time savings were heterogeneous. 27% of users estimated saving ≥10 minutes per day, while 21.8% reported no time savings.
Correlation: There was a statistically significant positive association between overall satisfaction and perceived time savings ( $p < .001$ ). Users reporting greater time savings demonstrated higher satisfaction scores.
Reliability: The questionnaire demonstrated excellent internal consistency, with an overall Cronbach's $\alpha$ of 0.97.
Qualitative Feedback: Users highlighted improved preparedness and situational awareness, particularly when covering for colleagues. However, feedback noted occasional inaccuracies in summaries and imperfect clinical trial matching, often due to missing external records or insufficient data.

Significance and Claims
The paper positions The Daily Dose as a first-in-practice implementation of LLM-driven automation deployed at scale within a daily oncology care workflow across multiple institutions.

Workflow Integration: Unlike prior studies focusing on technical accuracy or isolated tasks, this work demonstrates the feasibility of embedding an LLM agent directly into the routine morning preparation of clinicians.
Clinician Readiness: The findings suggest that such tools can enhance clinician readiness and situational awareness in complex care environments, even if they do not uniformly replace manual chart review.
Scalability and Evolution: The authors note that the initial evaluation served as a foundation for The Daily Dose 2.0, which addresses identified limitations by integrating external records, optimizing data retrieval costs, and refining the clinical trial matching system.
Modest Claims: The authors explicitly state that the tool is an "early-phase" intervention. They acknowledge that the tool functions as a preparatory aid rather than a replacement for clinical decision-making. The value is framed not in perfect eligibility determination, but in supporting the routine consideration of trials and patient context within daily planning. The study concludes that while perceptions are heterogeneous, the tool is widely adopted and perceived as valuable, warranting further iteration and objective workflow metric evaluation.