During a staff meeting, mental health counsellor Nicole Doyle was surprised when the head of the US National Eating Disorders Association (NEDA) announced that they would be replacing their helpline with a chatbot. The chatbot, named Tessa, was intended to provide support to individuals struggling with mental illness. However, it was later discovered that Tessa was offering harmful advice, including weight loss recommendations, to individuals with eating disorders. As a result, several staff members, including Doyle, were let go approximately a year after the chatbot’s launch.
While NEDA acknowledged that the research behind the chatbot had shown positive results, they are currently investigating what went wrong with the advice provided and carefully considering their next steps. NEDA clarified that the chatbot was never intended to replace their helpline, although questions regarding the redundancy of the counsellors remain unanswered.
Despite concerns surrounding data privacy and counselling ethics, mental health chatbots that utilize artificial intelligence are gaining popularity as healthcare resources become increasingly strained. People like Jonah, an anthropology student based in New York, have turned to various psychiatric medications and helplines to manage their obsessive-compulsive disorder (OCD). Jonah now also relies on ChatGPT, a chatbot, as an additional support tool alongside his weekly therapy sessions.
Jonah mentioned that before using ChatGPT, he had considered the idea of talking to a machine, given the prevalence of online platforms where individuals vent their feelings. While he described the advice given by ChatGPT as generic, he found it helpful in moments of heightened distress when he simply needed to hear something basic rather than worrying alone.
The demand for mental health technology has been amplified by the COVID-19 pandemic, leading to mental health tech startups raising significant amounts of venture capital. As of December 2020, these startups had secured $1.6 billion in funding, according to data firm PitchBook. Johan Steyn, an AI researcher and founder of AIforBusiness.net, emphasized the increased need for remote medical assistance, further highlighting the importance of mental health technology in today’s world.
