ChatGPT Shows Low Accuracy in Pediatric Diagnoses, Revealing Need for Improvement in AI Medical Applications

A recent study published in JAMA Pediatrics highlighted the limitations of ChatGPT in diagnosing pediatric medical cases. The fourth version of the large language model achieved only a 17% accuracy rate in this domain, significantly lower than its general diagnostic accuracy rate of 39% reported last year. Conducted by researchers at Cohen Children’s Medical Center in New York, the study tested ChatGPT-4 against 100 pediatric case challenges from JAMA Pediatrics and NEJM between 2013 and 2023.

The AI model’s performance in these cases was evaluated by two qualified physician-researchers. ChatGPT correctly diagnosed only 17 out of 100 cases. In 72 instances, it was completely wrong, and in the remaining 11, it identified a clinically related condition that was too broad or unspecific. For example, ChatGPT misdiagnosed a case of Branchio-oto-renal syndrome as a branchial cleft cyst.

This outcome suggests that despite the rapid advancement in AI technology, human pediatricians remain indispensable, especially given the specific challenges in diagnosing children, who often cannot fully express their symptoms. The researchers observed that ChatGPT struggled with identifying connections between conditions, a skill that comes with clinical experience. For instance, it failed to link autism with scurvy (Vitamin C deficiency), a connection that is well-known among experienced clinicians.

The study’s authors suggest that ChatGPT’s diagnostic capabilities could be enhanced by training it on more accurate and trustworthy medical literature and providing it with real-time access to medical data for ongoing refinement and tuning. This approach could potentially improve the accuracy of AI chatbots in complex diagnostics.