Author(s)
John M. Sommerfeldt, MD
James J. Pierre, MD
Jakob L. Fischer, MD
Charles A. Riley, MD
Anthony M. Tolisano, MD
Affiliation(s)
Tripler Army Medical Center
Abstract:
Educational Objective: At the conclusion of this presentation, the participants should be able to understand the role that artificial intelligence systems such as ChatGPT can play in enhancing patient counseling for complex and difficult to understand diagnoses.
Objectives: To compare the quality of Google, ChatGPT, and expert providers in answering patient questions regarding vestibular schwannoma.
Study Design: Observational.
Methods: 10 commonly asked patient questions regarding vestibular schwannoma were generated based on search engine frequency data. Answers to the questions were then generated from Google search, ChatGPT, and expert provider. Response quality was graded by three blinded, independent board certified otolaryngologists using DISCERN scores, after which total scores, subsection scores and readability scores were compared. Inter-rater agreement was determined using Fleiss-Kappa score.
Results: Inter-rater agreement was high with a Fleiss' Kappa score of 0.73. Total DISCERN scores between Google, ChatGPT, and expert response were 23.7, 30.4, and 28.4, respectively, with statistically significant differences between both ChatGPT and expert response compared with Google. The average word count for ChatGPT was 213.9 words, which was significantly higher than Google (68.8) and expert response (108.2), though had the highest readability score with a Flesch Reading Ease score of 33.1.
Conclusions: ChatGPT generated fair quality information in response to questions regarding diagnosis and management of vestibular schwannoma, which was not significantly different from human expert responses. The overall quality of response was improved with treatment related questions as opposed to questions not related to treatment. While concern often exists regarding the quality of artificial intelligence generated information, the results of this study demonstrate that ChatGPT can be a useful tool for patient counseling for vestibular schwannoma.