💻 computer science

DR. INFO at the Point of Care: A Prospective Pilot Study of Physician-Perceived Value of an Agentic AI Clinical Assistant

This prospective pilot study demonstrates that physicians across various specialties perceived the agentic AI assistant "DR. INFO" as highly effective for improving time efficiency and clinical decision support, yielding a high Net Promoter Score despite the need for further validation in larger, controlled trials.

Original authors: Rogerio Corga Da Silva, Miguel Romano, Tiago Mendes, Marta Isidoro, Sandhanakrishnan Ravichandran, Shivesh Kumar, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rogerio Corga Da Silva, Miguel Romano, Tiago Mendes, Marta Isidoro, Sandhanakrishnan Ravichandran, Shivesh Kumar, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a doctor's day as a marathon where the runner is constantly trying to read a library of books while running, answering patient questions, and filling out endless paperwork. This is the reality for many physicians today: they spend more time wrestling with computers and searching for information than actually talking to patients. This leads to exhaustion and burnout.

Enter DR. INFO, a new digital assistant powered by Artificial Intelligence (AI). Think of it not as a "magic oracle" that knows everything instantly, but as a super-powered research librarian who is incredibly fast, never sleeps, and has memorized millions of medical textbooks and journals.

Here is the story of a small pilot study that tested this librarian in the real world, explained simply:

The Mission: A "Test Drive" for Doctors

The researchers wanted to see if doctors and medical students felt that DR. INFO actually saved them time and helped them make better decisions. They didn't just ask, "Do you like it?" they asked, "Does it make your job easier?"

They gathered 29 participants (mostly doctors, some students) in Portugal. These weren't tech novices; they were already used to using digital tools. They were given a "test drive" of the software for five working days. Every day, they used the tool to help with real patient cases and then filled out a quick survey, like a customer feedback card at a restaurant.

The Results: A Standing Ovation (With a Few Critiques)

The feedback was overwhelmingly positive. Here's what the "customers" said:

Time Saver: On a scale of 1 to 5, doctors rated the time-saving aspect a 4.27. Imagine asking a team of researchers to find a specific fact in a library; usually, it takes them 20 minutes. DR. INFO did it in seconds. Doctors felt they got their time back.
Decision Support: They rated the help with medical decisions a 4.16. It was like having a wise mentor whispering, "Hey, have you checked this guideline?" right when they needed it.
The "Recommendation" Score (NPS): If you asked these doctors, "Would you recommend this to a friend?" the score was 81.2. In the world of customer satisfaction, anything over 50 is considered "World Class." It's the difference between a "good" product and a "must-have" product.

Who liked it most?
Interestingly, the younger doctors and students seemed to love it the most (scoring it nearly perfect), while senior, experienced doctors liked it too, though slightly less enthusiastically. It's like a new sports car: the new drivers are thrilled by the speed, while the veterans appreciate the safety features but are a bit more cautious.

The "But..." (The Critiques)

No tool is perfect. When asked what they disliked, the doctors pointed out two main issues:

Speed: Sometimes the librarian was a little slow. Because DR. INFO is designed to be safe, it doesn't just guess; it goes to the library, finds the book, reads it, and checks the source before answering. This "double-checking" took a few extra seconds, which felt like an eternity to a busy doctor.
Too General: Sometimes the answers were a bit like a Wikipedia summary—correct, but not specific enough for a very complex, unique patient case.

The Safety Net: How It Works

The most important part of this story is how DR. INFO works.
Many AI tools are like a confident student who might make up facts if they don't know the answer (called "hallucinations").
DR. INFO is different. It is an "Agentic" AI.

Analogy: Imagine a detective. A regular AI might guess who the culprit is based on a hunch. DR. INFO is a detective who must find the physical evidence, check the police report, and verify the witness before writing the report.
It doesn't write prescriptions or make decisions for the doctor. It just gathers the facts and cites its sources. The doctor remains the captain of the ship, making the final call.

The Catch: It's Still a Pilot

The researchers are very honest about the limitations. This was a small "test drive," not a full-scale race.

Small Group: Only 29 people tried it.
Self-Reported: We only know what the doctors felt. We didn't measure if patients actually got better (though that's the goal).
No "Truth" Check: The study didn't have a panel of experts verify if every single answer the AI gave was 100% medically perfect (though a separate technical test showed it was very accurate).

The Bottom Line

This study is like a proof of concept. It shows that when you give doctors a smart, safe, and source-checking AI assistant, they love it. They feel it saves them time and helps them think clearer.

The researchers conclude that while the tool is promising, we need bigger, more rigorous tests to make sure it works perfectly for everyone, everywhere. But the first step? It looks like doctors are ready to welcome this new digital librarian into their team.

1. Problem Statement

Physicians face significant cognitive overload and burnout, with administrative tasks and clinical documentation consuming over 50% of their working hours. While Artificial Intelligence (AI) and Large Language Models (LLMs) offer potential solutions, their adoption in clinical settings is hindered by safety concerns, specifically hallucinations (factually incorrect but plausible outputs) and a lack of source reliability. General-purpose LLMs often operate as "black boxes" without access to curated medical knowledge, making them unsuitable for high-stakes decision-making. There is a critical need for agentic AI systems that can retrieve, verify, and synthesize information from trusted medical databases before generating responses, thereby bridging the gap between available evidence and bedside practice.

2. Methodology

The study was a prospective, single-arm pilot feasibility study conducted in October 2025 across Portuguese healthcare institutions.

System Under Study: DR. INFO v1.0, an agentic AI clinical assistant.
- Architecture: Unlike standalone LLMs, DR. INFO uses a Retrieval-Augmented Generation (RAG) approach. It decomposes physician queries, performs clinical reasoning, retrieves and verifies sources from curated medical knowledge bases and peer-reviewed literature, and synthesizes cited responses.
- Constraints: It does not access Electronic Health Records (EHR), generate prescriptions, or perform autonomous actions. The physician retains full decision-making authority.
Participants: 29 participants (25 physicians and 4 medical students) across various specialties (Family Medicine, ICU, Internal Medicine, Surgery, etc.) and experience levels.
Protocol:
- Participants used the tool for five working days over a two-week period.
- Data Collection: Three electronic Case Report Forms (eCRFs) were used:
  1. Baseline: Demographics, years of practice, and technology comfort.
  2. Daily Diary: Likert-scale ratings (1–5) for Time Saving and Decision Support, plus qualitative use-case logging.
  3. Final Evaluation: Net Promoter Score (NPS) and open-ended feedback.
Statistical Analysis: Due to the small sample size and exploratory nature, non-parametric methods were employed (Mann-Whitney U, Friedman test, Spearman's rank correlation). Confidence intervals were calculated using bootstrap resampling (10,000 iterations). Sensitivity analyses were performed to address the 45% attrition rate in the final evaluation.

3. Key Contributions

Agentic AI Validation: This study provides early empirical evidence on the utility of an agentic clinical assistant (which verifies sources) versus a standard generative LLM in a real-world clinical workflow.
Technical Benchmarking Context: The paper contextualizes DR. INFO against the HealthBench (a 5,000-case clinical benchmark). On the "Hard" subset, DR. INFO scored 0.68, significantly outperforming standalone models like GPT-5 (0.40–0.46) and Grok 3 (0.23), demonstrating the value of the retrieval architecture over raw model capability alone.
User-Centric Design: The system was developed via a "co-design" process with a multi-specialty "Innovator Circle," ensuring the tool addresses actual clinical workflows rather than theoretical use cases.
Attrition Bias Analysis: The study rigorously analyzed the impact of the 45% non-response rate on the final NPS, providing a conservative lower-bound estimate of user satisfaction.

4. Key Results

Perceived Time Efficiency: Physicians reported high perceived time savings with a mean score of 4.27/5 (95% CI: 3.97–4.57). 87.8% of diary entries indicated agreement or strong agreement.
Decision Support: The tool received a mean score of 4.16/5 (95% CI: 3.86–4.45) for supporting clinical reasoning (diagnosis, treatment, information retrieval). 85.6% of entries agreed it improved decisions.
Stability: Scores remained stable across the five-day study window (Friedman test: $p > 0.05$ ), suggesting no significant "novelty effect" decay during the short trial.
Net Promoter Score (NPS):
- Among the 16 completers, the NPS was 81.2 (81% Promoters, 19% Passives, 0% Detractors).
- Sensitivity Analysis: Even under conservative assumptions where all 13 non-respondents were classified as "Passives," the NPS remained positive at 44.8. If all were "Detractors," the NPS would drop to 0.0.
Demographic Trends:
- Junior vs. Senior: There was a moderate, non-significant inverse correlation between years of practice and perceived decision support ( $\rho = -0.381, p=0.055$ ), suggesting junior physicians and students may derive slightly higher relative value.
- Tech Comfort: No correlation was found between self-reported technology comfort and perceived utility ( $\rho = 0.060$ ), indicating the interface is accessible regardless of technical proficiency.
Qualitative Feedback:
- Top Use Cases: Treatment advice (47 entries), disease details/differential diagnosis (46), and medication details (36).
- Primary Criticisms: Response latency (50% of feedback) due to the time required for source retrieval/verification, and insufficient specificity of answers (38%).

5. Significance and Limitations

Significance:
The study demonstrates that an agentic AI assistant, designed with source verification and human-in-the-loop constraints, is perceived by physicians as a valuable tool for improving time efficiency and clinical decision-making. The high NPS and stable scores suggest that such tools can be integrated into routine workflows without disrupting existing practices. The results support the hypothesis that retrieval-augmented architectures are superior to standalone LLMs for clinical safety and utility.

Limitations:

Lack of Objective Accuracy Verification: The study relied on perceived value. It did not independently verify the clinical correctness of the AI's specific responses generated during the pilot.
Single-Arm Design: Without a control group or direct comparison to other tools (e.g., UpToDate), the results reflect perceived utility rather than measured incremental performance.
Sample Size and Attrition: The small sample (N=29) and 45% attrition rate limit statistical power and generalizability.
Ceiling Effects: High baseline satisfaction with existing tools may have limited the ability to detect nuanced differences in the Likert scale.
Short Duration: The 5-day window cannot rule out long-term novelty effects or fatigue.

Conclusion:
While preliminary, the findings are promising and hypothesis-generating. They justify the need for larger, controlled studies that include objective accuracy auditing and longitudinal performance measures to confirm that perceived benefits translate into measurable improvements in patient safety and clinical outcomes.