Journal Shows Its Commitment to Exploring AI in Medicine

The New England Journal of Medicine is charging ahead with its coverage of the impact of artificial intelligence (AI) in medicine, publishing a special report, a review, and an editorial in this week’s issue, and announcing the creation of a journal dedicated to AI.

The special report, written by researchers from Microsoft, outlined potential medical uses of a generative AI chatbot powered by GPT-4 (generative pretrained transformer 4), including physician note taking, medical education, and participation in “curbside consults.”

GPT-4 achieved an accuracy of about 90% when given a battery of test questions from the U.S. Medical Licensing Exam (USMLE), even though it was only trained on openly available information from the internet and never received specific medical training, Peter Lee, PhD, of Microsoft Research in Redmond, Washington, and co-authors reported.

When tasked with writing a medical note from only a provider-patient interaction, GPT-4 performed well but it did misstate the patient’s body-mass index and contains no information as to how the BMI was calculated, “another example of a hallucination,” the researchers wrote.

They also provide an example of how a physician may interact with GPT-4 on a medical consult, in which it “generally provides useful responses that may help the health professional who made the query address the problem of concern.”

While GPT-4 could be a powerful tool in both clinical settings and medical research, the researchers said, they emphasized the need for careful consideration for any potential uses of GPT-4 in medicine, adding that “GPT-4 is an intelligent system that, similar to human reason, is fallible.”

“We believe that the question regarding what is considered to be acceptable performance of general AI remains to be answered,” they wrote. “Our hope is to contribute to what we believe will be an important public discussion about the role of this new type of AI, as well as to understand how our approach to health care and medicine can best evolve alongside its rapid evolution.”