AI technology was found to be impressively accurate at diagnosing patients in health care settings, according to a new study.
The artificial intelligence (AI) chatbot ChatGPT was found to be 72 percent accurate in clinical decision-making in health care settings, the study published in the Journal of Medical Internet Research found. The software was able to come up with diagnoses for patients, and even provide the next steps for care.
"Our paper comprehensively assesses decision support via ChatGPT from the very beginning of working with a patient through the entire care scenario, from differential diagnosis all the way through testing, diagnosis, and management," corresponding author Marc Succi, MD, said in a press release.
"No real benchmarks exists, but we estimate this performance to be at the level of someone who has just graduated from medical school, such as an intern or resident. This tells us that LLMs in general have the potential to be an augmenting tool for the practice of medicine and support clinical decision making with impressive accuracy," said Succi, the associate chair of innovation and commercialization and strategic innovation leader at Mass General Brigham and executive director of the MESH Incubator.
It was also found that the chatbot performed just as well in different health care environments, from primary care and emergency care.
To reach these findings, researchers at the Mass General Brigham asked the AI system for various diagnoses based on patient information feed into it. The researchers also included specific details such as patient gender, age and symptoms.
While the bot was 72 percent accurate in clinical decision-making, it performed the best in final diagnoses, where it was 77 percent accurate.
It did experience some setbacks. Researchers found it was less accurate (60 percent) at making several possible diagnoses. With other decisions such as assigning medications depending on the diagnosis, it was also less effective, but still 68 percent accurate.
"ChatGPT struggled with differential diagnosis, which is the meat and potatoes of medicine when a physician has to figure out what to do," Succi said. "That is important because it tells us where physicians are truly experts and adding the most value—in the early stages of patient care with little presenting information, when a list of possible diagnoses is needed."
Although it could never replace human professionals in this way, the growth of AI technology could change the way many industries look in a few years' time. Although its ability to assist industries like health care needs to be tested more thoroughly, this study suggests that it could be implemented in the future.
"Mass General Brigham sees great promise for large-language models (LLMs) to help improve care delivery and clinician experience," co-author Adam Landman, the chief information officer and senior vice president of digital at Mass General Brigham said.
"We are currently evaluating LLM solutions that assist with clinical documentation and draft responses to patient messages with focus on understanding their accuracy, reliability, safety, and equity. Rigorous studies like this one are needed before we integrate LLM tools into clinical care."
Do you have a tip on a science story that Newsweek should be covering? Do you have a question about AI? Let us know via science@newsweek.com.
Uncommon Knowledge
Newsweek is committed to challenging conventional wisdom and finding connections in the search for common ground.
Newsweek is committed to challenging conventional wisdom and finding connections in the search for common ground.
About the writer
Robyn White is a Newsweek Nature Reporter based in London, UK. Her focus is reporting on wildlife, science and the ... Read more
To read how Newsweek uses AI as a newsroom tool, Click here.
