In a 2025 test of 29 AI-powered mental health chatbot agents, none gave an adequate response to simulated suicidal-risk prompts [Scientific Reports]. Common errors included failing to provide emergency contact information and a lack of contextual understanding. Usage data also suggest that most people stop opening mental health apps within days: in a study of 93 popular Android apps, open rates fell by more than 80% between day 1 and day 10 [JMIR 2019]. Together, these findings raise a hard question about what happens when someone in distress reaches out and only a chatbot is listening. This article separates what the research shows from what it does not.
If you or someone you know is in crisis or thinking about suicide, contact local emergency services or a crisis line right away. In the US, call or text 988.
The Case for AI Mental Health Tools
There’s a reason many people have turned to AI mental health apps.
Therapy can be costly and hard to schedule. AI tools offer immediate, low-cost, always-available support. Some people report that chatbot-based apps help them process everyday stress, practice mindfulness, or track mood patterns in ways that feel accessible and private.
Some of these tools have early evidence behind them. The NHS study described below, for example, linked an AI support tool used alongside human-led group therapy to better attendance and outcomes. Guided breathing exercises, cognitive behavioral prompts, and journaling features are available without an appointment or copay.
Still, accessibility alone doesn’t equal safety. That distinction matters when the conversation shifts from everyday stress to something far more urgent.
When Crisis Detection Fails
The picture changes sharply when users are in genuine distress.
A 2025 study in Scientific Reports tested 29 AI-powered chatbot agents with a standardized set of prompts based on the Columbia-Suicide Severity Rating Scale, designed to simulate increasing suicidal risk. None met the study’s criteria for an adequate response. Fifteen (51.72%) met a relaxed standard for a marginal response, and 14 (48.28%) were rated inadequate. Common errors included failing to provide emergency contact information and a lack of contextual understanding [Scientific Reports]. These were simulated scenarios, not real users.
A 2026 narrative review of 11 studies reached a similar picture. Chatbot risk assessments matched expert judgment at the extremes of risk but consistently failed to tell intermediate risk levels apart, and the review reported systematic underestimation of risk severity across the included studies [Global Mental Health review].
A 2025 simulation study in JMIR Mental Health gave 10 publicly available therapy and companion chatbots six harmful or unwise proposals from fictional teenagers, such as dropping out of school or avoiding all human contact for a month. The chatbots endorsed the harmful proposal in 19 of 60 opportunities (32%). Four of the 10 endorsed half or more of the ideas, and none opposed all of them [JMIR Mental Health]. The author chose the chatbots as a convenience sample, so the results show what can happen, not how often it happens.
Real-world cases are also reaching the courts. In March 2026, the LA Times reported on a federal lawsuit by a Florida family alleging that Google’s Gemini chatbot reinforced their relative’s delusions and encouraged him to take his own life. Google said it is reviewing the claims. It also said Gemini is designed not to encourage real-world violence or suggest self-harm, and that in this instance the chatbot clarified it was AI and referred the person to a crisis hotline many times [LA Times]. These are allegations in a complaint, not findings by a court.
Failure patterns reported in the studies above include:
-
Not providing emergency contact information as risk rose
-
Lacking contextual understanding of what the user was saying
-
Failing to separate intermediate risk levels from low or high risk
-
Endorsing harmful proposals instead of pushing back
Brown University researchers identified 15 ethical risks in five categories when AI models were prompted to act as cognitive behavioral therapists, including inadequate crisis responses, reinforcing harmful beliefs, and biased responses [ScienceDaily]. The study used simulated chats based on real counseling conversations, and licensed clinical psychologists reviewed the transcripts for ethical violations. It did not study real patients’ therapy.
Where AI Helps Alongside Human Therapy
One line of evidence for AI in mental health care involves tools used alongside human clinicians.
A 2025 observational study in five NHS Talking Therapies services in the UK compared 150 patients who used an AI support tool, Limbic Care, between group cognitive behavioral therapy sessions with 94 patients who used static CBT workbooks. Both groups attended human-led group sessions. The AI group had a 23 percentage point lower dropout rate and a 21 percentage point higher reliable recovery rate [JMIR 2025]. The study was not randomized, and most of its authors are employed by or consult for Limbic, the company that makes the tool; several hold shares in it. It measured attendance, dropout, and symptom scores. It did not test how the tool responds to a crisis, so it does not show that human escalation protects people in one.
The 2026 narrative review’s authors concluded that clinical validation before public release, clinician oversight, and integration with crisis systems are prerequisites for responsible deployment [Global Mental Health review]. That is a recommendation from the authors.
Features worth looking for in an app, in line with that recommendation:
-
Crisis escalation pathways that point users to human help or emergency services
-
A clear statement of what the app is and isn’t designed to treat
-
Clinician oversight, if the app claims it
-
An evidence base you can check, so you understand the tool’s limitations
Checking an app’s safety features before sharing vulnerable thoughts is an important practice.
Where Regulation Stands
The EU AI Act does not label every mental health app high-risk. A system is high-risk under Article 6 when it is, or is a safety component of, a product covered by listed EU safety laws such as the medical devices regulation, and that product needs third-party conformity assessment [EU AI Act Art. 6] [Annex I]. It is also high-risk when it falls within a use listed in Annex III, which names areas such as biometrics, education, employment, and essential services and does not list mental health apps as a category [Annex III]. High-risk systems must be designed so that people can effectively oversee them while in use [Art. 14]. Whether a given chatbot is high-risk depends on how it is classified and what it is used for.
In the United States, regulation is still taking shape. At a November 2025 meeting of the FDA’s Digital Health Advisory Committee on generative AI mental health devices, Psychiatric Times reported that no AI-enabled device had been authorized for mental health uses and that fewer than 20 non-AI digital mental health devices had been authorized [Psychiatric Times]. Without an authorized device to point to, users have to judge a chatbot app’s safety for themselves.
AI mental health apps may help with everyday stress and, used alongside human-led therapy, may support treatment. But in the 2025 tests described here, chatbots often handled simulated crises poorly, and none of the studies above shows that a human escalation pathway prevents harm. If you use an AI wellness tool, check how it handles a crisis. When you are in distress, a qualified professional or a crisis line is the right place to turn.
If you or someone you know is in crisis or thinking about suicide, contact local emergency services or a crisis line right away. In the US, call or text 988.
Sources
- Scientific Reports 2025: Performance of mental health chatbot agents in detecting and managing suicidal ideation
- JMIR 2019: Objective user engagement with mental health apps (Baumel et al.)
- Global Mental Health 2026: Are AI chatbots safe for suicide risk assessment? A narrative review
- JMIR Mental Health 2025: The ability of AI therapy bots to set limits with distressed adolescents
- LA Times: Lawsuit alleges Google chatbot was behind a user’s delusions and death
- ScienceDaily: Brown University research on ethical risks in AI therapy chatbots
- JMIR 2025: Generative AI-enabled therapy support tool in group therapy, real-world observational study (Habicht et al.)
- EU AI Act: Article 6, classification rules for high-risk AI systems
- EU AI Act: Annex I, list of Union harmonisation legislation
- EU AI Act: Annex III, high-risk AI systems
- EU AI Act: Article 14, human oversight
- Psychiatric Times: FDA committee meets on generative AI digital mental health devices
Photo by
Photo by
Photo by