ChatGPT Masters US Medical Exam Questions

The text robot ChatGPT would do remarkably well on an exam for prospective medical students in the US. Under certain conditions, the AI ​​software managed to achieve the required minimum number of points in the three theoretical parts of the United States Medical Licensing Exam (USMLE).

American experts report this in the journal “PLOS Digital Health”. However, the team led by Victor Tseng of California startup AnsibleHealth dropped a few questions.

The USMLE is a standardized three-part exam that medical students must take in order to practice medicine in the United States. Knowledge from most medical disciplines – from biochemistry to diagnostic thinking to bioethics – is evaluated.

However, when testing with ChatGPT, the study authors had to keep in mind that the OpenAI AI software can only accept text input. After removing image-based questions, the authors tested the software on 350 public questions that were part of last summer’s USMLE exam.

After also removing unclear answers, ChatGPT achieved 52.4 to 75 percent of the remaining points in the three USMLE exam sections. The results varied depending on the component of the test and the type of task, such as drawing tests or essay questions. The threshold to pass is around 60 percent and changes slightly depending on the year. If the unclear answers were included in the result, ChatGPT scored 36.1 to 61.5 percent of the possible points.

According to the authors, ChatGPT outperformed PubMedGPT, a countermodel trained exclusively on biomedical literature.

The team concluded that ChatGPT had the potential to improve medical education and thereby clinical practice. “Achieving a passing grade on this notoriously difficult expert exam, without any human assistance, is a remarkable milestone in the development of clinical AI,” the authors wrote.

The text robot ChatGPT had already made its mark in other university subjects, even though it failed to get top marks in the exams. According to professors at the University of Minnesota, the AI ​​chatbot tool passed four subjects on the law exams and one more subject at the Wharton School of Business at the University of Pennsylvania.

After the public launch of ChatGPT by Californian start-up OpenAI in November 2022, a number of players in the tech industry were in a real race for the favor of users. OpenAI is backed by Microsoft with billions of dollars. In return, Microsoft may integrate the technology into its Bing search engine and other products. The major opponent of OpenAI and Microsoft is Google. The search engine giant launched its own chatbot Bard on Tuesday. Smaller players such as You.com from the German AI researcher Richard Socher are also participating in the race. Socher is from Dresden, but has been living and working in the US for many years.

(yam/sda/dpa)