Trust in, acceptance of, and responsibility for, artificial intelligence in healthcare: patient and healthcare practitioner considerations

This qualitative study comparing healthcare practitioners and patients in a Dutch surgical oncology setting reveals that while both groups share concerns about AI responsibility, their perspectives diverge significantly, with clinicians focusing on clinical validation and integration while patients prioritize human connection and trust.

The Gist

Imagine the healthcare system as a massive, busy airport. For years, the pilots (doctors) and the ground crew (nurses) have been making decisions about when planes (patients) can take off (go home) based on their experience, gut feelings, and checklists.

Now, a new piece of technology has arrived: a super-smart, high-tech flight computer (Artificial Intelligence) that can process millions of flight logs in a second to predict the perfect takeoff time.

This research paper is like a group of pilots, ground crew, and passengers sitting down to discuss: "Do we trust this new computer? Who is really in charge of the plane? And how do we make sure everyone feels safe?"

Here is a simple breakdown of what they found, using everyday analogies.

1. The Knowledge Gap: "The Black Box" vs. "The Magic Wand"

• The Doctors & Nurses: They know the computer is powerful, but they are wary of the "Black Box." They don't want to just press a button and hope for the best. They want to know: Where did this computer learn its rules? Is it looking at the right data? They worry that if they don't understand how the computer thinks, they might blindly follow a bad suggestion.
• The Patients: Most patients are like people who see a magic wand on TV. They know AI is "a thing" that's popular, but they don't know how it works. They aren't necessarily scared of the magic, but they are scared of being tricked by it. They want to know: Is this computer looking at my specific story, or just a generic story?

The Takeaway: You can't just hand a pilot a new computer and say "fly." You have to teach them how to read the screen, and you have to tell the passengers what the computer is actually doing.

2. The "Human Touch" vs. The "Algorithm"

• The Doctors: They worry about "Automation Bias." Imagine a GPS that says "Turn Left," but you see a giant wall. If you blindly follow the GPS, you crash. Doctors fear they might stop using their own "gut feelings" (like knowing a patient looks tired even if their numbers look fine) because the computer says "Go."
• The Patients: They are worried about being treated like a spreadsheet. They said, "The computer doesn't know how my body feels." They fear that if the computer says "You can go home," the doctor might just nod and send them away without asking, "But how are you really feeling?"

The Takeaway: The computer is a great assistant, but it can't feel the wind or see the fear in a passenger's eyes. The human needs to stay in the loop to catch what the machine misses.

3. Who is the Captain? (Responsibility)

• The Doctors: They are very clear: "I am the Captain." Even if the computer says "Land now," the doctor has to make the final call. If the plane crashes, the doctor is the one who has to answer for it. They feel a heavy weight of responsibility.
• The Patients: They mostly said, "I trust the Captain." They don't care who programmed the computer; they just want to know that the doctor is paying attention. However, some patients worried that if the computer says "Go," the doctor might use it as an excuse to stop listening to the patient's complaints.

The Takeaway: The doctor must remain the Captain. The computer is just the co-pilot giving advice. If the co-pilot is wrong, the Captain must be brave enough to say, "No, we are staying."

4. Trust: The "Friend" vs. The "Tool"

• The Doctors: They don't trust the computer yet. They want to see it work perfectly for a long time before they fully rely on it. They want to test it, monitor it, and check its work constantly.
• The Patients: They don't trust the computer at all—and that's okay. They trust their doctor. If their doctor says, "This computer is a good tool, and I've checked it, so I'm using it to help you," then the patient is happy. If the doctor says, "The computer says go," without explaining, the patient gets nervous.

The Takeaway: Patients trust the person holding the tool, not the tool itself.

5. The "Seamless" Problem (Workflow)

• The Reality: Doctors are already drowning in paperwork. They are terrified that this new AI will be like a new app that requires them to fill out 50 extra forms just to get one piece of advice.
• The Wish: They want the AI to be like a smart thermostat that just works in the background. They want the advice to pop up on their screen automatically, without them having to log into a new system or click a hundred boxes.

The Takeaway: If the new technology makes the doctor's job harder or slower, they won't use it, no matter how smart it is. It has to be invisible and helpful.

The Big Conclusion

The study found that while doctors and patients often talk about the same things (safety, fairness, trust), they see them through different lenses.

• Doctors are worried about technical errors and losing their professional judgment.
• Patients are worried about being ignored and losing their human connection with their doctor.

The Final Lesson: To make AI work in healthcare, we can't just build a smarter computer. We have to build a system where:

1. The computer is transparent (we know how it thinks).
2. The doctor stays in charge (the human is the Captain).
3. The patient feels heard (the human connection isn't replaced by a screen).

If we get the technology right but forget the human feelings, the "flight" will still crash.

Technical Summary

1. Problem Statement

Despite significant investment and the "hype" surrounding Artificial Intelligence (AI) in healthcare since the early 2020s, a substantial implementation gap persists. Studies indicate that only a small fraction (e.g., ~2%) of AI models successfully transition from research to the bedside. Current literature often focuses on technical performance or implementation logistics, neglecting the human and ethical dimensions of adoption. Specifically, there is a lack of empirical research comparing the perspectives of healthcare practitioners (HCPs) and patients regarding AI integration. This gap is critical because the successful deployment of AI relies not just on algorithmic accuracy, but on trust, acceptance, and the clear allocation of responsibility within the clinical epistemic environment.

2. Methodology

• Study Design: An exploratory qualitative study utilizing semi-structured interviews.
• Setting: Department of Surgical Oncology and Gastrointestinal Surgery at Erasmus Medical Center, Rotterdam, The Netherlands.
• Participants ($N=30$):
  • Healthcare Practitioners (18): 10 surgeons and 8 nurses (purposively sampled to include those responsible for daily patient care).
  • Patients (12): Individuals who underwent gastrointestinal surgery within the preceding 12 months.
  • Exclusion: Patients with major health complications were excluded.
• Data Collection: Interviews lasted 30–45 minutes. HCPs were interviewed in-person; patients were interviewed primarily online (Microsoft Teams).
• Data Analysis:
  • Transcripts were analyzed using ATLAS.ti.
  • A hybrid approach of deductive and inductive coding was employed.
  • Two first authors independently coded subsets of data, compared codebooks to resolve discrepancies, and iteratively refined a shared codebook to identify thematic categories.
  • Thematic saturation was reached when no new themes emerged.

3. Key Contributions

The study makes three primary contributions to the field of AI in healthcare:

1. Comparative Stakeholder Analysis: It uniquely juxtaposes the views of clinicians and patients using identical interview guides, revealing where perspectives converge and diverge.
2. Contextualized Findings: The study examines perceptions of a specific, validated AI model (the DESIRE model for post-operative discharge decisions), enhancing the practical relevance of the findings compared to abstract theoretical discussions.
3. Ethical and Epistemic Framework: It moves beyond technical metrics to analyze the epistemic environment of healthcare, highlighting how trust, responsibility, and autonomy are negotiated between humans and algorithms.

4. Key Results

The analysis identified three main thematic categories with ten sub-themes:

A. AI Knowledge

• Literacy Gap: Both groups demonstrated broad awareness of AI's societal presence but varied significantly in technical depth. Many HCPs and most patients described their knowledge as "surface-level" or media-informed.
• Implication: Limited technical literacy did not lead to rejection; rather, participants viewed understanding limitations (uncertainty, error rates) as a prerequisite for safe trust.
• Divergence: Clinicians focused on validation and data sources; patients focused on reassurance and the need for accessible explanations.

B. Ethics

• Autonomy:
  • HCPs: Focused on preserving professional judgment and "human-in-the-loop" control to avoid automation bias. They valued "tacit knowledge" (intuition/gut feeling) which AI cannot replicate.
  • Patients: Focused on bodily autonomy and the right to be heard. They feared AI might silence their subjective experiences or strengthen paternalistic doctor-patient dynamics.
• Responsibility & Accountability:
  • Consensus: Both groups agreed the physician bears ultimate responsibility for decisions.
  • Nuance: HCPs expressed tension between accepting responsibility for tools they did not fully trust. Patients increasingly recognized the distributed nature of responsibility (involving developers, hospitals, and systems), whereas HCPs viewed it more hierarchically.
• Explainability:
  • HCPs: Valued explainability for practical decision-making (knowing inputs/limitations) rather than full algorithmic transparency ("black box" acceptance was common if outputs were actionable).
  • Patients: Contrary to HCP assumptions that patients wouldn't care, many patients expressed a desire for transparency regarding how AI influenced their care, linking it to trust and agency.
• Bias and Fairness:
  • HCPs: Viewed bias as a technical/data issue (training set representativeness, generalizability).
  • Patients: Viewed bias through a social lens, concerned about specific demographic groups (e.g., ethnicity) being disadvantaged.

C. Operational and Clinical Implications

• Clinical Impact: Generally positive views on AI as a decision-support tool to handle data volume and reduce uncertainty. However, a "wait-and-see" attitude prevailed regarding tangible benefits.
• Workflow Integration: Success is contingent on seamless integration into Electronic Health Records (EHR). Participants feared AI would increase administrative burden if it required extra clicks or separate portals.
• Technical Concerns: Major barriers included data quality, the need for continuous monitoring (models degrade over time), and the difficulty of capturing nuanced clinical factors (e.g., pain, mobility) in structured data.

5. Significance and Conclusion

The study concludes that the adoption of AI in healthcare is not merely a technical challenge but a socio-technical and ethical one.

• Trust Dynamics: Clinicians trust AI based on validation and monitoring, while patients trust AI only mediated through their trust in the physician.
• Epistemic Injustice: There is a risk that AI outputs could be used by clinicians as "evidence" to override patient concerns, potentially silencing patients and creating epistemic injustice.
• Recommendations:
  1. Targeted Education: Structured training is needed for both HCPs and patients to bridge literacy gaps and manage expectations.
  2. Communication Strategies: AI communication cannot be "one-size-fits-all"; it must be adaptable to patient preferences for information.
  3. System Design: AI tools must be integrated seamlessly into clinical workflows to avoid increasing administrative load.
  4. Ethical Governance: Responsibility frameworks must address the distributed nature of AI accountability, ensuring patients remain active participants in their care rather than passive subjects of algorithmic decisions.

Ultimately, responsible AI deployment requires cultivating an environment where clinicians maintain critical oversight and patients feel their subjective experiences are valued alongside algorithmic predictions.