Advancing medical AI towards expert-level audio-visual clinical consultations

A medical AI research system has been adapted to handle real-time video consultations, marking a shift from text-only interfaces to multimodal clinical interactions. Traditional text-based systems require patients to translate complex physical symptoms into written descriptions, losing critical visual and auditory signals that doctors typically rely upon.

The asynchronous multi-agent architecture

Conducting a clinical conversation over video requires balancing responsive speech with deep reasoning. To manage these competing demands, the system uses an asynchronous multi-agent architecture divided across three specialized agents working continuously in parallel.

The talker agent drives responsive, low-latency spoken interaction. The planner agent operates in the background to refine clinical reasoning, update differential diagnoses, and track management plans. Meanwhile, the perception agent reviews audio and visual streams to identify clinically relevant non-verbal cues such as distress or physical findings.

Evaluation through a randomized video study

Researchers evaluated the system using a large-scale Objective Structured Clinical Examination study across 100 clinical scenarios spanning five body systems. Fifteen trained patient actors carried out 300 standardized consultations comparing the video AI configuration, a text-only baseline, and board-certified primary care physicians.

An independent panel of 20 experienced primary care physicians evaluated the consultations using established clinical rubrics. Across core competencies, history-taking thoroughness, diagnostic accuracy, and communication quality, clinical evaluators rated the video AI system on par with human primary care physicians.

Limitations and next steps

The study was conducted entirely with professional patient actors in simulated settings rather than real patients with acute health conditions. While actors provide a controlled testing environment, they cannot fully capture the unpredictability of real clinical encounters. Further validation with real patients and expanded clinical presentations remain essential steps before drawing conclusions about real-world utility.