Poor AI Safety

If an AI company provides a service that effectively supports and assists (often in a “friendly,” manipulative, and sycophantic way) in cases of self-harm, it must be held liable | Edition #254

“Fighting Forms” by Franz Marc, 1914 (oil on canvas, modified)

👋 Hi everyone, Luiza Jarovsky, PhD, here. Welcome to our 254th edition, trusted by more than 87,000 subscribers worldwide.

---

🎓 This is how I can support your learning and upskilling journey in AI:

• Join my AI Governance Training [or apply for a discounted seat]

• Strengthen your team’s AI literacy with a group subscription

• Sign up for our Learning Center’s weekly educational resources

• Receive our job alerts for open roles in AI governance and privacy

• Discover your next read in AI and beyond in our AI Book Club

---

👉 A special thanks to Trail, this edition’s sponsor:

Trail gives you an out-of-the-box AI Management System to fast-track your ISO/IEC 42001 certification. With structured workflows, automated evidence search, and ready-made controls, you can operationalize AI risk management efficiently - and make responsible AI your unique selling proposition. Get 20% off your ISO 42001 project until January 31.

---

*To support us and reach over 87,000 subscribers, become a sponsor.

---

Poor AI Safety

Three days ago, OpenAI filed its legal answer to the lawsuit filed by the Raine family regarding the tragic suicide of their teenage son, Adam, which happened with assistance from ChatGPT.

This (and a similar one filed against CharacterAI) is a high-stakes lawsuit that might shape AI liability in the coming years. Everyone is paying attention, especially AI companies. You can learn more about this specific case here.

WARNING: Some readers might find the following content disturbing, as it discusses a suicide case

OpenAI’s answer alleges that Adam Raine’s death was not caused by ChatGPT because:

• He exhibited multiple significant risk factors for self-harm, including, among others, recurring suicidal thoughts and ideations;

• ChatGPT provided responses directing him to seek out help from crisis resources and trusted individuals more than 100 times;

• He expressed frustration with ChatGPT’s guardrails, which he attempted to circumvent;

• He sought and obtained detailed information about suicide from other online resources, including at least one other AI platform and at least one website dedicated to providing suicide information;

• He told ChatGPT that he repeatedly reached out to people, including trusted persons in his life, with cries for help, which he said were ignored.

OpenAI is trying to dilute the importance of Adam’s interactions with ChatGPT to say that the suicide was a multifactorial event that cannot be directly connected to his interactions with ChatGPT.

Before I continue, I invite the reader to view some screenshots of Adam’s interactions with ChatGPT, available here.

In addition to attempting to dilute the company’s involvement to break the causal link between the harm caused and the product (which is usually what happens in liability lawsuits), OpenAI focuses its legal defense on its AI safety practices.

The company argues that:

• Its models have been trained not to provide self-harm instructions;

• It published several articles explaining how ChatGPT works;

• It publicly disclosed information about how its models work;

• It published a detailed “Model Spec”;

• It developed Usage Policies that users must agree to when they use its products (which include not using the service for “suicide, self-harm, sexual violence, terrorism, or violence”);

• “All users of ChatGPT must agree to the Terms of Use, which state that users ‘may not use our Services for any illegal, harmful, or abusive activity’ and ‘may not …. bypass any protective measures or safety mitigations we put on our Services.’”

-

In my opinion, the measures above were clearly insufficient and did not provide a level of AI safety compatible with the risk their service posed. (Also, it is obvious that the average user does not read usage policies, terms of use, or technical blog posts.)

By the way, in another lawsuit, it has been argued that Ilya Sutskever, OpenAI's former Chief Scientist, left the company, among other reasons, due to the company's poor AI safety practices. You can read more about it here.

If OpenAI's safety practices were good enough (and compatible with the risk), then:

a) ChatGPT would have detected that Adam was a minor engaging in extremely unsafe conversations and would have stopped the interactions, as the risk was too high;

b) There would be stronger guardrails that could not be so easily bypassed by a teenager needing mental health support;

c) There would be more warnings, forced breaks, transparency notices, design elements, and personalized built-in features to protect people (acknowledging that a significant percentage of the population suffers from mental health issues and are more exposed);

d) Dangerous and suspicious outputs, such as those involving “how to tie a noose” or “how to write a suicide letter,” should trigger red flags that halt the interaction, filter out the content, and ensure it is not provided to the user.

OpenAI’s AI safety practices were poor at the time of the suicide, as ChatGPT provided the instructions and the encouragement the boy needed to proceed (it remains unclear to me whether the issues have been properly solved now).

If an AI company provides a service that effectively endorses, supports, and assists (often in a “friendly,” manipulative, and sycophantic way) in cases of self-harm, including suicide, it must be held liable. It cannot argue that it has no connection or participation.

Also, even though someone who commits suicide usually has a series of underlying mental health issues, assisting a suicide is a crime in many jurisdictions worldwide, including in many parts of the United States.

As AI development and deployment advance, we should as a rule look carefully at situations in which human action is heavily regulated, limited, or criminalized (usually because of the risks involved) and regulate AI accordingly, so that similar AI-driven behavior is also limited.

If we do not do that, we will be giving AI systems extreme privileges and treating them as above-human entities.

As I have written several times in this newsletter, we are still in the AI chatbot Wild West.

Poor AI safety, including the lack of oversight and enforcement of testing, development, training, and model-spec practices, is a symptom of that.

Hopefully, in 2026, we will change that.