💬 NLP

Towards Expert-level Medical AI for Real-time Video Consultations

This paper introduces AMIE (Video), a Gemini-based multi-agent system that achieves expert-level performance in real-time clinical video consultations by integrating audio-visual perception and clinical reasoning, demonstrating in a randomized study with physicians and patient actors that it matches or exceeds human doctors in diagnostic accuracy and history-taking while offering a preferred interface for patient communication.

Original authors: Mahvish Nagda, Jihyeon Lee, Matthew Thompson, Chunjong Park, Tim Strother, Valentin Liévin, Roma Ruparel, Akshay Goel, Teya Bergamaschi, Suhana Bedi, Meet Shah, Pavel Dubov, Liviu Panait, Toshiyuki Fu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mahvish Nagda, Jihyeon Lee, Matthew Thompson, Chunjong Park, Tim Strother, Valentin Liévin, Roma Ruparel, Akshay Goel, Teya Bergamaschi, Suhana Bedi, Meet Shah, Pavel Dubov, Liviu Panait, Toshiyuki Fukuzawa, Sam Schmidgall, Craig Schiff, Joseph Xu, Aliya Rysbek, Yana Lunts, Jan Freyberg, Rebecca Hemengway, Sunny Virmani, David Racz, Carey Radebaugh, Joëlle Barral, Kavi Goel, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Gregory Wayne, Tao Tu, Yun Liu, Ethan Goh, Christina Chen, Ryutaro Tanno, Po-Hsuan Cameron Chen, Mike Schaekermann, Anil Palepu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your doctor isn't just someone you talk to, but someone who can see you, too. For a long time, artificial intelligence (AI) in medicine has been like a very smart librarian who can only read books. It's great at answering questions if you type them out, but it can't see your pale face, hear the wheeze in your breath, or notice you're tapping your foot nervously. This is a big problem because human doctors don't just listen to words; they watch your posture, your facial expressions, and how you move to figure out what's wrong. They use these visual clues to build trust and make better guesses about your health. But teaching a computer to do this in real-time, while you're having a video call, is like trying to teach a robot to juggle while riding a unicycle on a tightrope. It needs to be fast, it needs to see everything, and it needs to think deeply about what it sees, all at the same time.

This paper introduces a new kind of medical AI called AMIE (Articulate Medical Intelligence Explorer) that tries to master this tightrope act. Instead of just reading your typed messages, AMIE (Video) watches and listens to you through a video call, just like a real doctor would. The researchers wanted to see if this AI could act like an expert doctor during a real-time video consultation. They didn't just ask the AI to guess; they put it to the test in a giant simulation called an OSCE (Objective Structured Clinical Examination). In this test, professional actors played the role of patients with various health problems, and they had video calls with either the AI or real, board-certified primary care doctors. The goal was to see who could ask the right questions, spot the right physical signs, and come up with the best plan to help the patient.

Here is what they found: The AI was surprisingly good. In fact, in many ways, it performed just as well as, or even better than, the human doctors. When independent experts graded the consultations, the AI scored higher than the human doctors in taking a patient's history, making a diagnosis, planning treatment, and guiding the patient through physical exams over the video. The AI was particularly good at using the video feed to "see" things, like asking a patient to turn their head or raise their arms to check for pain or stiffness. The human actors playing the patients also liked the AI's video version much more than a text-only chat; they felt the video interface was easier to use and made them feel more understood.

However, the story isn't a fairy tale where the robot has completely replaced the human. While the AI was a superstar at the medical facts and the physical checks, the human doctors still won the popularity contest for building a warm, friendly connection. The actors felt the human doctors were better at building rapport and making them feel like partners in their care. The AI also had some "glitches" in its superpowers. It sometimes missed very subtle things, like a tiny tremor in a hand or a fleeting change in a person's mood, and it struggled with high-speed movements. Also, the study was a simulation with actors, not real patients in a real clinic, so while the results are a huge milestone, the AI isn't ready to replace your doctor just yet. It's more like a brilliant new intern who is amazing at the technical work but is still learning how to be a warm, empathetic human being. The paper suggests that with more work, this technology could one day help doctors everywhere by handling the heavy lifting of observation and diagnosis, freeing them up to focus on the human connection.

Technical Summary: Towards Expert-level Medical AI for Real-time Video Consultations

1. Problem Statement

Clinical encounters rely heavily on non-verbal cues—vocal tone, facial expressions, posture, and physical signs like tremors or dyspnea—which are foundational to clinical reasoning, trust, and therapeutic outcomes. While text-based medical AI has demonstrated expert-level performance in static benchmarks and text-only conversations, it discards these essential perceptual dimensions. Text interfaces force patients to distill complex physical and behavioral observations into discrete descriptions, introducing bias and potentially omitting diagnostic information. Furthermore, text-only systems present barriers to equitable access for patients with lower digital or health literacy.

Early attempts to extend medical AI to audio-visual interaction have shown feasibility but have not yet reached clinician-level performance. A significant challenge lies in balancing low-latency dialogue (necessary for natural conversation) with thorough clinical reasoning and real-time audio-visual perception. Fully characterizing these systems requires large-scale evaluation, including modality ablations and granular analyses of clinical audio-visual capabilities.

2. Methodology

System Architecture: AMIE (Video)

The authors introduce AMIE (Articulate Medical Intelligence Explorer) in a video configuration, built upon Gemini 3 Flash and Gemini 3.1 Pro. The system employs a novel asynchronous multi-agent architecture comprising three specialized agents designed to overcome latency and perception challenges:

Talker Agent: The user-facing interface responsible for generating fluent, low-latency audio responses. It processes the most recent 5 seconds of video frames and synthesizes inputs from the other agents. It manages conversational turn-taking, listening for interruptions and transitioning to silent listening mode when necessary.
Planner Agent: Manages clinical goals, maintaining a persistent memory of the patient's state, differential diagnosis, and management plan. It asynchronously updates clinical goals (e.g., eliciting specific symptoms, instructing physical maneuvers) and synthesizes patient information. Updates are decoupled from the Talker to ensure conversational fluency.
Perception Agent: Dedicated to processing continuous audio and video streams. It enumerates clinically relevant visual and auditory information (e.g., affect, speech cadence, physical signs) and maintains a cumulative memory of all detected cues. This agent addresses the limitation of the Talker's restricted reasoning budget by reasoning over extended video windows.

Evaluation Framework

To guide development, the authors established a taxonomy of clinical audio-visual cues feasible in telehealth settings, derived from medical literature. This taxonomy categorizes cues by "telehealth feasibility" and acquisition method (passive vs. guided).

The evaluation framework includes:

Single-turn Evaluations: "Unit tests" isolating specific capabilities (e.g., identifying anatomical location or laterality) using audio/video clips and LLM-based auto-raters.
Multi-turn Simulated Consultations: Assessing end-to-end conversational capability using injected verbal stage directions to simulate visual cues.

The OSCE Study

A randomized, multi-arm Objective Structured Clinical Examination (OSCE) was conducted to compare performance:

Participants: 15 professional patient actors (enacting 100 clinical scenarios) and 10 board-certified Primary Care Physicians (PCPs).
Study Arms:
1. AMIE (Video): Real-time video consultation.
2. AMIE (Text): Text-only chat interface.
3. PCP (Video): Human physicians consulting via video (cameras off to match the avatar-less AI).
Evaluation: Independent clinical evaluators (20 PCPs) rated consultations using general and case-specific rubrics covering history-taking, diagnosis, management, physical observation, and communication. Patient actors also rated their experience regarding rapport, empathy, and modality preference.

3. Key Contributions

AMIE (Video) System: The first demonstration of an AI system achieving expert-level performance in real-time clinical video consultations. The system successfully integrates low-latency dialogue with deep clinical reasoning and continuous audio-visual perception via an asynchronous multi-agent design.
Comprehensive Evaluation Framework: The development of a telehealth-specific taxonomy and an automated evaluation suite that bridges single-turn capability testing with multi-turn simulated consultations.
Rigorous Clinical Benchmarking: A large-scale, randomized OSCE study comparing AI (in both video and text modalities) against human physicians across 100 diverse clinical scenarios.

4. Results

Performance vs. Primary Care Physicians (PCPs)

In the video consultation study, AMIE (Video) was rated on par or better than PCPs across core clinical competencies:

Overall Score: AMIE (Video) scored 83% compared to 68% for PCPs ( $p = 1.32 \times 10^{-9}$ ).
Diagnostic Accuracy: AMIE (Video) matched the correct reference diagnosis in its top differential item 91% of the time, compared to 77% for PCPs ( $p = 0.039$ ).
Clinical Reasoning & Treatment Planning: AMIE (Video) significantly outperformed PCPs in clinical reasoning (90% vs. 76%) and treatment planning (79% vs. 67%).
Physical Observation & Examination: AMIE (Video) scored significantly higher in utilizing live video for information gathering (77% vs. 51%) and guiding physical exams (72% vs. 39%).
History Taking: Rated equivalent to or better than PCPs (86% vs. 78%).

Patient Preferences: Patient actors preferred AMIE (Video) for assessing and explaining conditions. However, PCPs were preferred (though not statistically significantly) for rapport and partnership building.

Modality Ablation (Video vs. Text)

Comparing AMIE (Video) to AMIE (Text):

Patient Preference: Actors significantly preferred the video interface for communicative effectiveness, ease/convenience, and feeling understood.
Clinical Performance: Clinical evaluators rated AMIE (Video) higher than AMIE (Text) on overall case-specific rubrics (83% vs. 75%), with the largest gains in perception and examination. Diagnostic accuracy and management appropriateness were comparable between modalities.

Automated Evaluation Insights

Agent Contributions: The full multi-agent system (59% accuracy in single-turn tests) significantly outperformed a Talker-only baseline (42%). The Perception Agent provided the largest individual boost, particularly for visual inspection tasks. The Planner Agent improved reasoning on complex tasks like gait instability.
Latency: The asynchronous architecture reduced mean turn latency from ~21.4s (prior work) to 2.6s.
Limitations: The system showed gaps in perceiving subtle affective nuances, high-frequency movements (e.g., tremors, nystagmus), and fine anatomical precision.

5. Significance and Claims

The paper claims this work represents a milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.

Expert-Level Performance: This is the first demonstration of an AI system matching or exceeding human physician performance in a randomized, video-based clinical setting, moving beyond text-only or static image analysis.
Overcoming Text Limitations: By leveraging real-time audio-visual streams, the system bypasses the reporting biases and accessibility barriers inherent in text-based interfaces, capturing objective clinical data (e.g., respiratory effort, visual signs) that patients often struggle to articulate.
Architectural Innovation: The decoupled, asynchronous multi-agent design successfully resolves the tension between the need for rapid conversational turn-taking and the computational cost of deep clinical reasoning and continuous perception.
Ecological Validity: Unlike prior studies where human baselines were evaluated in text-only settings, this study evaluated AI against human physicians in the modality (video) they routinely use, providing a more realistic assessment of clinical performance.

Caveats and Limitations:
The authors explicitly state that the system is not ready for real-world clinical deployment. Key limitations include:

Inability to perform certain physical exams (e.g., palpation, fundoscopy) or detect subtle signs (e.g., tremors, nystagmus) reliably.
Reliance on patient actors rather than real patients, which limits the generalizability to unscripted behavioral and environmental complexities.
The study excluded dermatological conditions that cannot be authentically simulated and scenarios requiring privacy-sensitive exams.
The "human baseline" had limitations (e.g., physicians' cameras were off), which may have reduced the ecological validity of the human comparison regarding rapport building.

The authors conclude that while these results mark an important step, substantial validation in real clinical workflows is essential before responsible translation to practice.