Loading…
Report
Towards Democratization of Subspeciality Medical Expertise
O'Sullivan, Jack W., Palepu, Anil, Saab, Khaled, Weng, Wei-Hung, Cheng, Yong, Chu, Emily, Desai, Yaanik, Elezaby, Aly, Kim, Daniel Seung, Lan, Roy, Tang, Wilson, Tapaskar, Natalie, Parikh, Victoria, Jain, Sneha S., Kulkarni, Kavita, Mansfield, Philip, Webster, Dale, Gottweis, Juraj, Barral, Joelle, Schaekermann, Mike, Tanno, Ryutaro, Mahdavi, S. Sara, Natarajan, Vivek, Karthikesalingam, Alan, Ashley, Euan, Tu, Tao
Saved in:
| Title | Towards Democratization of Subspeciality Medical Expertise |
|---|---|
| Authors | O'Sullivan, Jack W., Palepu, Anil, Saab, Khaled, Weng, Wei-Hung, Cheng, Yong, Chu, Emily, Desai, Yaanik, Elezaby, Aly, Kim, Daniel Seung, Lan, Roy, Tang, Wilson, Tapaskar, Natalie, Parikh, Victoria, Jain, Sneha S., Kulkarni, Kavita, Mansfield, Philip, Webster, Dale, Gottweis, Juraj, Barral, Joelle, Schaekermann, Mike, Tanno, Ryutaro, Mahdavi, S. Sara, Natarajan, Vivek, Karthikesalingam, Alan, Ashley, Euan, Tu, Tao |
| Publication Year |
2024
|
| Description |
The scarcity of subspecialist medical expertise, particularly in rare, complex and life-threatening diseases, poses a significant challenge for healthcare delivery. This issue is particularly acute in cardiology where timely, accurate management determines outcomes. We explored the potential of AMIE (Articulate Medical Intelligence Explorer), a large language model (LLM)-based experimental AI system optimized for diagnostic dialogue, to potentially augment and support clinical decision-making in this challenging context. We curated a real-world dataset of 204 complex cases from a subspecialist cardiology practice, including results for electrocardiograms, echocardiograms, cardiac MRI, genetic tests, and cardiopulmonary stress tests. We developed a ten-domain evaluation rubric used by subspecialists to evaluate the quality of diagnosis and clinical management plans produced by general cardiologists or AMIE, the latter enhanced with web-search and self-critique capabilities. AMIE was rated superior to general cardiologists for 5 of the 10 domains (with preference ranging from 9% to 20%), and equivalent for the rest. Access to AMIE's response improved cardiologists' overall response quality in 63.7% of cases while lowering quality in just 3.4%. Cardiologists' responses with access to AMIE were superior to cardiologist responses without access to AMIE for all 10 domains. Qualitative examinations suggest AMIE and general cardiologist could complement each other, with AMIE thorough and sensitive, while general cardiologist concise and specific. Overall, our results suggest that specialized medical LLMs have the potential to augment general cardiologists' capabilities by bridging gaps in subspecialty expertise, though further research and validation are essential for wide clinical utility.
|
| Document Type |
Working Paper
|
| Subject Terms |
