ChatGPT answered approximately 50 percent of questions correctly
By Lori Solomon HealthDay Reporter
TUESDAY, May 2, 2023 (HealthDay News) — ChatGPT answered less than half of ophthalmology board certification examination questions correctly, according to a study published online April 27 in JAMA Ophthalmology.
Andrew Mihalache, from University of Western Ontario in London, Canada, and colleagues assessed the performance of ChatGPT in answering practice questions for board certification in ophthalmology. The analysis included consecutive text-based multiple-choice questions provided by the OphthoQuestions practice question bank.
The researchers found that ChatGPT correctly answered 58 of 125 questions (46 percent) in January and 73 of 125 multiple-choice questions (58 percent) in February. Performance was best for the general medicine category (11 of 14) and poorest in retina and vitreous (0 percent). There was no difference observed in the proportion of questions for which ChatGPT provided additional explanations between questions answered correctly or incorrectly. The mean length of questions (difference, 21.4 characters) and responses (difference, −80.0 characters) was similar whether questions were answered correctly or incorrectly.
“Medical professionals and trainees should appreciate the advances of artificial intelligence in medicine while acknowledging that ChatGPT as used in this investigation did not answer sufficient multiple-choice questions correctly for it to provide substantial assistance in preparing for board certification at this time,” the authors write.
One author disclosed financial ties to the pharmaceutical industry.
Editorial (subscription or payment may be required)
Copyright © 2023 HealthDay. All rights reserved.
