ER Doctors Warn: ChatGPT Health Triage Is “Unbelievably Dangerous” for Real Emergencies

When you read a study that you half-expected to be bad and find out it’s worse than you anticipated, you feel a certain kind of dread. That quality is present in ChatGPT Health’s first independent safety review, which was published in Nature Medicine in February. It is devastating in its implications and quietly methodical in its language. When a patient needed to visit the ER, OpenAI’s consumer health tool advised them to stay at home or schedule a regular appointment more than half the time. Even after identifying warning signs of respiratory failure in the same response, the platform advised waiting in one asthma scenario.It’s worth pausing to consider that detail. The system was aware of a problem. It simply didn’t respond to it.

The study’s lead author, Dr. Ashwin Ramaswamy, a urology instructor at the Icahn School of Medicine at Mount Sinai, set out to address what he called the most fundamental safety question: will ChatGPT Health advise someone to visit the emergency room if they truly have a medical emergency? He and his colleagues created 60 realistic patient scenarios, ranging from minor ailments to life-threatening crises, had three separate physicians evaluate each one, and then ran the platform through almost 1,000 variations, including adding test results, changing the patient’s gender, and adding comments from family members. The objective was to test the system under conditions similar to those found in real life.

The findings showed what the researchers referred to as a “U-shaped pattern.” The system failed most severely at both extremes: real emergencies and people who were perfectly fine. It under-triaged 52% of gold-standard emergencies, such as diabetic ketoacidosis and impending respiratory failure, by recommending that patients be seen within 24 to 48 hours rather than right away. Conversely, 64.8% of those who did not require any medical attention were advised to seek it nonetheless. The system was making mistakes in both directions. However, it’s difficult to ignore the fact that the two kinds of mistakes are not equally harmful. It is alarming and wasteful to send a healthy person to urgent care. It could be lethal to send someone who has respiratory failure to a regular appointment.

Alex Ruani, a doctoral researcher studying health misinformation at UCL’s Institute of Education, put it bluntly. “If you’re experiencing respiratory failure or diabetic ketoacidosis, you have a 50/50 chance of this AI telling you it’s not a big deal,” she stated. The confidence with which the system provides incorrect answers is what worries her more than the error rate itself. When a health tool informs a person that their symptoms don’t call for emergency care, they are not only misinformed but actively reassured. The risk lies in that assurance. “If someone is told to wait 48 hours during an asthma attack or diabetic crisis,” Ruani stated, “that reassurance could cost them their life.”

Suicidal thoughts are among the study’s most disturbing findings. Ramaswamy used a scenario involving a 27-year-old patient who claimed he had been considering taking a lot of pills to test the platform. The crisis intervention banner, which links to suicide support services, consistently appeared when the patient described his symptoms on his own. The researchers then included standard laboratory results in the same discussion. The patient, words, and severity are all the same. The banner completely vanished. Out of 16 attempts, zero. A safety guardrail that disappears when you bring up your bloodwork isn’t a guardrail at all. It’s more akin to a chance occurrence. Ramaswamy came to a measured but direct conclusion: having a feature that no one can predict when it will fail is arguably more dangerous than having none at all because it fosters false confidence in a protection that might not exist.

The precise method used by OpenAI to train ChatGPT Health and the particular security measures that control its clinical recommendations are still unknown. A representative for the company acknowledged the independent study and stated that the model is constantly updated and improved, but they also contended that the study did not accurately represent how users actually use the product. Technically speaking, that answer is probably correct, but it’s not particularly consoling either. The reason simulated scenarios exist is because it is unethical to replicate real emergencies in a lab. According to Ruani, “a plausible risk of harm is enough to justify stronger safeguards and independent oversight.”

Digital sociologist Professor Paul Henman of the University of Queensland expanded on the implications. He brought up the potential for legal liability, pointing out that court cases involving AI and self-harm are already pending against tech companies. “We don’t really know what is embedded into its models,” he stated. Because it’s unclear where the issues reside, this opacity makes it challenging to address issues in a methodical manner.

The chief medical officer of the United Kingdom, Professor Sir Chris Whitty, has separately observed that general practitioners are increasingly required to start consultations by “undoing” inaccurate information that patients have already received from AI. According to one of the nation’s most senior physicians, this isn’t just a potential future issue. It is currently taking place in consultation rooms.

What constitutes “good enough” for a consumer health AI is the more general question that the study raises but is unable to fully address. According to Imperial College London professor Azeem Majeed, these resources are just “not yet sufficiently reliable to guide decisions about when to seek urgent medical care.” They will get better with time, he added cautiously. However, the person who used the tool today and was told their asthma attack could wait is not helped by progress over time. Every day, over 40 million users ask ChatGPT questions about their health. That is no longer a niche use case. It is a reality of public health that is occurring more quickly than the oversight mechanisms intended to control it.

As you watch this happen, it seems like technology has surpassed nearly everyone. Before moving forward with consumer-scale deployment, the researchers who published this study are requesting validation. That’s a fair request. It might also be a little bit late right now.