Quality of information on Wilms tumor from artificial intelligence chatbots: What are your patients and their families reading?

Abstract

Objective To assess the ability of AI chatbots to deliver quality and understandable information on Wilms tumors to patients and their families. Methods Google trends were used to evaluate the most asked questions related to Wilms tumor. Four AI chatbots (ChatGPT version 3.5, Perplexity, Chat Sonic, and Bing AI) were then used to assess these questions and their responses reviewed. Validated instruments were used to assess the quality (DISCERN instrument from 1 low to 5 high), understandability and actionability (PEMAT, from 0% to 100%), the reading level of the information and whether there was misinformation compared to guidelines (5-point Likert scale). Results All AI chat bots provided a high level of patient health information with a median DISCERN score of 4 (IQR 3-5). Additionally, there was little to no misinformation in outputs with a median of 1 (IQR 1-1). The median word count per output from the AIs was 275 (IQR 156-322), with an advanced ease of reading level comparable to a high school or college student, median Flesch-Kincaid Readability level of 46.7 (IQR 41.1-52.2). The overall PEMAT actionability was poor with a median of 40% (40-65), while the PEMAT understandability of the AI chatbot outputs was high, 83% (IQR 75-91.2). Conclusion AI chatbots provide generalized, understandable and accurate information regarding Wilms tumor. They can be reliably used as a source for patients and families when seeking further information. However, much of the information is reliant of medical professionals and not easily actionable by consumers but may act as a guide to help with discussions and understanding treatments.