Israel Medical Association Journal

Background: Large language models (LLMs) are rapidly advancing, with the potential to improve healthcare. While LLM performance on medical licensing exams were studied extensively, their performance in rheumatology exams requires specific evaluation.

Objective: To assess Chat Generative Pre-Trained Transformer (ChatGPT) performance on 200 validated Israeli rheumatology board exam questions.

Methods: ChatGPT performance was evaluated using 200 multiple-choice questions from the 2023 and 2024 Israeli official rheumatology board examinations. Three gpt-4-turbo based variants were assessed: base model (Model 1), few-shot chain of thought (CoT) model (Model 2), and knowledge-augmented prompting model incorporating rheumatology guidelines (Model 3). Model 1 was assessed using both the original Hebrew and a validated English translated version, while Models 2 and 3 were assessed using the English version only.

Results: Overall, Model 3 achieved the highest numerical accuracy (81%), followed by Model 1 (English, 77%), Model 2 (75%), and Model 1 (Hebrew, 74.5%); however, these differences were not statistically significant. Performance varied markedly by question type. For text-only questions (n=177), accuracies ranged from 78.5% to 83.1%, with Model 3 showing the highest point estimate (83.1%). In contrast, all models demonstrated substantially lower performance on questions that included images (n=23), with accuracies ranging from 34.8% to 65.2%. Model 3 yielded the highest numerical accuracy (65.2%).

Conclusions: The study highlights the potential role of LLMs in rheumatology board examinations but also emphasizes their critical limitations. Future research should focus on addressing limitations, especially image interpretation and management of complex cases to enable efficient application of LLMs in rheumatology.

Background: Behcet's syndrome (BS) is a multisystem syndrome that typically manifests as recurrent oral and genital ulcers, as well as other systemic manifestations. Few studies describing the characteristics of BS among Israeli patients have been published.

Objectives: To describe the characteristics of BS patients and to compare Jewish and Arab subpopulations.

Methods: We retrospectively reviewed electronic medical records and extracted demographic, clinical, laboratory, and medication data for each patient. We compared the Jewish and Arabic BS patients.

Results: The cohort included 98 patients. Males constituted 49 (50%); mean age at the time of diagnosis was 29.9 years; 71 (72.4%) were Arab and 27 (27.6%) were Jewish. Oral ulcers were evident in 93 patients (94.9%) and genital ulcers in 54 (55.1%). Involvement of the skin, joints, eyes, gastrointestinal tract, and neurologic and vascular systems were demonstrated among 42 (42.9%), 57 (58.2%), 47 (48.0%), 8 (8.2%), 10 (10.2%), and 15 (15.3%), respectively. HLA B51 was positive in 24 of 37 (64.9%). Pathergy was positive in 8 of 12 (66.7%). Colchicine was used in 82 (83.7%), azathioprine 47 (48%), methotrexate 16 (16.3%), apremilast 10 (10.2%), cyclosporine-A 8 (8.2%), adalimumab 26 (26.5%), infliximab 12 (12.2%), cyclophosphamide 1 (1.0%), tocilizumab 2 (2.0%), and anti-coagulation 6 (6.1%). The Arab and Jewish subpopulations were significantly different regarding male proportion, 40 (56.3%) vs. 9 (33.3%), P = 0.042.

Conclusions: BS is more common among Arabs in northern Israel, but no significant clinical or demographic differences were found except for a higher proportion of male patients among Arabs.

The issue of genital ulcers (GUs) in Israeli Arab female patients with Behçet's disease (BD is important as GUs are considered a unique feature of BD and stand as diagnostic criterion for the diagnosis in the International Study Group (ISG) criteria [1] and the International Criteria for Behcet Disease (ICBD) [2]. According to the ISG, GUs can be detected by a physician or by the patient [1]. While oral ulcers are easily observable, the identification of more hidden GUs pose a significant challenge.

Since the introduction of the international study group (ISG) criteria for the diagnosis of Behçet's disease (BD) in the early 1990s by Yazici and colleagues [1] and the international criteria for BD (ICBD) by Davatchi and colleagues in 2014 [2], great progress has been achieved in the diagnosis of BD with fairly high sensitivity and specificity rates. However, a small, but very challenging and unique minority might not fulfill these criteria, at least at presentation. These patients are most challenging as they may present with life-threatening vascular or neurological manifestations. If the diagnosis BD is delayed, the risk for morbidity and even mortality might be increased. Therefore, we should aim for early diagnosis and prompt treatment.