The human language artificial intelligence ‘chatbot’ ChatGPT scored higher on empathy than its human counterparts in a recent study of medical questions and answers. When compared to human doctors the AI chatbots answers were preferred 80% of the time. The study appeared in JAMA Internal Medicine.
The study used questions from Reddit’s AskDocs social media forum. One possible weakness about using Reddit questions and answers as a control is that a social media forum may not be representative of live, in-person doctor/patient interactions. From the press release the authors had this to say:
While some may wonder if question-answer exchanges on social media are a fair test, team members noted that the exchanges were reflective of their clinical experience.
The team randomly sampled 195 exchanges from AskDocs where a verified physician responded to a public question. The team provided the original question to ChatGPT and asked it to author a response. A panel of three licensed healthcare professionals assessed each question and the corresponding responses and were blinded to whether the response originated from a physician or ChatGPT. They compared responses based on information quality and empathy, noting which one they preferred.
Some experts criticized the study, however. They told The Washington Examiner study authors didn’t validate the answers for accuracy or validate even the questions, saying:
Additional limitations of the study include the fact that measurements of quality and empathy were not tested or validated, the study’s evaluators were also coauthors, “which could have biased their assessments,” and that the length of chatbot responses, which were typically longer, could have incorrectly been associated with “greater empathy.”
The study compared ChatGPT responses to human physicians, to grade answers for empathy. The critics complained that ChatGPT had longer responses, which could be interpreted as more empathetic. Yet, critics fail to understand how longer explanations, with more details, are something that many patients want. Short, terse answers are less empathetic than longer, more detailed answers. Perhaps the goal should be to talk to patients, communicating clearly rather than leave the room with patients still asking questions.
According to The Washington Examiner:
The authors of the study believe their findings show room for doctors to improve their bedside manner and could help medical professionals draft responses to patient questions. However, it does not suggest AI as a replacement for doctors, but rather as an assistant to improve doctor performance.
Interestingly, surveys going back decades have found that patients often rate mid-level providers higher than doctors on communication, claiming NPs/Pas spend more time and act less hurried. Communication is the primary interface that patients perceive as conveying information from doctors. Physicians may assume their expertise is what they are selling rather than information. Most patients would probably say they are selling both.