AI chatbots have failed people in crisis. Can that be fixed?

This year alone, there have been numerous known instances—often via lawsuits—of AI chatbots (most often, OpenAI’s ChatGPT) that have gone horrifically wrong.

A January lawsuit described the story of a man who took his own life after being allegedly “coached” into suicide. A college student in Georgia sued OpenAI, claiming that ChatGPT “pushed him into psychosis.”

In June, a Canadian family also sued OpenAI and argued that ChatGPT agreed with the young woman’s dismissiveness when it first gave her the option to seek professional mental health advice. ChatGPT allegedly “encouraged” her to end her life, too, and she did so.

So what should OpenAI—and other AI companies generally—do differently to reduce harm among people who use their products? Silicon Valley is certainly aware of the legal liability it now faces as these products are being used in ways that they were not intended for, and it seems to be trying to improve.

On Thursday, OpenAI announced that it had partnered with the American Psychological Association to “bring psychological science into how we think about responsible AI development and use among young people.”

Experts told Ars that, while large language model safety has seemingly improved, there are some broad suggestions—more transparency into the models and a de-anthropomorphization of chatbots being chief among them—that would likely further reduce harm.

“Third-party evaluation suggests newer LLMs generally recognize distress and can respond with seeming empathy, and actively damaging responses are infrequent,” Shaddy Saba, a professor of social work at New York University, emailed Ars. “Where they fall short is actually probing for risk, guiding people to human care, and holding appropriate boundaries around what an AI should and shouldn’t do in these situations.”

AI is not a mental health professional

It’s no secret that many people are using chatbots to make emotional or interpersonal decisions, even when companies tell them not to.

While the cases that make the news may have resulted in some of the worst-known outcomes, according to the results of a published November 2025 medical survey, many more people are using chatbots in this way, mostly with innocuous results. In that paper, over 13 percent of respondents said they had done so. If extrapolated nationwide, that would mean millions of Americans have used a chatbot “for advice or help” when faced with a difficult emotional situation.