Advancing medical AI towards expert-level audio-visual clinical consultations

Researchers have expanded the capabilities of the medical AI system AMIE to conduct real-time video consultations.

2 min read

Diagram illustrating AMIE guiding a patient through a video call and breaking down the multi agent system architecture and evaluation study.
An overview diagram showing AMIE conducting a real time video consultation and using a multi agent architecture for clinical interaction.

A medical AI research system has been adapted to handle real-time video consultations, marking a shift from text-only interfaces to multimodal clinical interactions. Traditional text-based systems require patients to translate complex physical symptoms into written descriptions, losing critical visual and auditory signals that doctors typically rely upon.

The asynchronous multi-agent architecture

Conducting a clinical conversation over video requires balancing responsive speech with deep reasoning. To manage these competing demands, the system uses an asynchronous multi-agent architecture divided across three specialized agents working continuously in parallel.

Illustration of a patient performing a physical movement in front of a laptop running the AMIE video consultation interface.
A patient actor interacting with the video AI system during a simulated clinical examination., Source ↗

The talker agent drives responsive, low-latency spoken interaction. The planner agent operates in the background to refine clinical reasoning, update differential diagnoses, and track management plans. Meanwhile, the perception agent reviews audio and visual streams to identify clinically relevant non-verbal cues such as distress or physical findings.

Evaluation through a randomized video study

Researchers evaluated the system using a large-scale Objective Structured Clinical Examination study across 100 clinical scenarios spanning five body systems. Fifteen trained patient actors carried out 300 standardized consultations comparing the video AI configuration, a text-only baseline, and board-certified primary care physicians.

Graphs showing diagnostic accuracy curves and clinical evaluator point ratings across multiple core competencies for AMIE versus physicians.
Charts displaying diagnostic accuracy and clinical evaluator ratings comparing AMIE and primary care physicians in video consultations., Source ↗

An independent panel of 20 experienced primary care physicians evaluated the consultations using established clinical rubrics. Across core competencies, history-taking thoroughness, diagnostic accuracy, and communication quality, clinical evaluators rated the video AI system on par with human primary care physicians.

Limitations and next steps

The study was conducted entirely with professional patient actors in simulated settings rather than real patients with acute health conditions. While actors provide a controlled testing environment, they cannot fully capture the unpredictability of real clinical encounters. Further validation with real patients and expanded clinical presentations remain essential steps before drawing conclusions about real-world utility.

Source: Original announcement

Worth a read?

Report

Comments