To assess ChatGPT’s performance on the AUA SASP and stratify performance by question stem complexity.
Questions from the 2021-2022 AUA SASP Program were administered to ChatGPT version 3 (ChatGPT-3). Questions were administered to the model utilizing a standardized prompt. The answer choice selected by ChatGPT was then used to answer the question stem in the AUA SASP program. ChatGPT was then prompted to assign a question stem order (first, second, third) to each question. The percentage of correctly answered questions were determined for each order level. All responses provided by ChatGPT were qualitatively assessed for appropriate rationale.
A total of 268 questions were administered to ChatGPT. ChatGPT performed better on the 2021 compared to 2022 AUA SASP question set answering 42.3% versus 30.0% of questions correctly (p 0.05).
ChatGPT answered many high level questions correctly and provided reasonable rationale for each answer choice. While ChatGPT was unable to answer numerous first order questions, future language processing model (LPM) learning may lead to optimization of its fund of knowledge. This may lead to the utilization of artificial intelligence like ChatGPT as an educational tool for urology trainees and professors.

Copyright © 2023 Elsevier Inc. All rights reserved.