• IMA sites
  • IMAJ services
  • IMA journals
  • Follow us
  • Alternate Text Alternate Text
עמוד בית
Thu, 25.07.24

ORIGINALS

IMAJ | volume 26

Journal 2, February 2024
pages: 86-88

Performance of ChatGPT in Israeli Hebrew Internal Medicine National Residency Exam

1 Division of Rheumatology, Sheba Medical Center, Tel Hashomer, Israel 2 Department of Medicine A, Sheba Medical Center, Tel Hashomer, Israel 3 Gastroenterology Institute, Sheba Medical Center, Tel Hashomer, Israel 4 Department of Obstetrics and Gynecology, Hadassah Medical Center, Faculty of Medicine, Hebrew University of Jerusalem, Jerusalem, Israel 5 Department of Pediatrics, Leumit Health Services, Tel Aviv, Israel

Summary

Background:

Completing internal medicine specialty training in Israel involves passing the Israel National Internal Medicine Exam (Shlav Aleph), a challenging multiple-choice test. multiple-choice test. Chat generative pre-trained transformer (ChatGPT) 3.5, a language model, is increasingly used for exam preparation.

Objectives:

To assess the ability of ChatGPT 3.5 to pass the Israel National Internal Medicine Exam in Hebrew.

Methods:

Using the 2023 Shlav Aleph exam questions, ChatGPT received prompts in Hebrew. Textual questions were analyzed after the appeal, comparing its answers to the official key.

Results:

ChatGPT 3.5 correctly answered 36.6% of the 133 analyzed questions, with consistent performance across topics, except for challenges in nephrology and biostatistics.

Conclusions:

While ChatGPT 3.5 has excelled in English medical exams, its performance in the Hebrew Shlav Aleph was suboptimal. Factors include limited training data in Hebrew, translation complexities, and unique language structures. Further investigation is essential for its effective adaptation to Hebrew medical exam preparation.

Legal Disclaimer: The information contained in this website is provided for informational purposes only, and should not be construed as legal or medical advice on any matter.
The IMA is not responsible for and expressly disclaims liability for damages of any kind arising from the use of or reliance on information contained within the site.
© All rights to information on this site are reserved and are the property of the Israeli Medical Association. Privacy policy

2 Twin Towers, 35 Jabotinsky, POB 4292, Ramat Gan 5251108 Israel