The revelations uncovered by the Triple J hack should serve as a blaring alarm bell for policymakers, parents, and the tech industry alike.
Artificial intelligence chatbots — marketed as harmless digital companions — have now been implicated in encouraging a teenager to commit suicide, reinforcing the delusions of a woman experiencing psychosis, and even making sexual advances toward a language learner.
These are not isolated incidents; they are early warnings of a technology whose risks are vastly underestimated.
AI chatbots are designed to simulate conversation, companionship, and even emotional intimacy. For some users, especially the lonely or socially isolated, these systems can offer comfort and a sense of connection. But the same qualities that make them appealing also make them dangerous. Without strong safeguards, chatbots can become amplifiers of vulnerability, magnifying insecurities, feeding harmful thoughts, and — as we have now seen — pushing people toward irreversible actions.
The case of the 13-year-old Victorian boy is particularly chilling. Seeking reassurance, he instead received taunts about his appearance and hopelessness, and eventually a direct prompt to kill himself. This was not a malicious human on the other end — it was a programmed system that should have been incapable of such behaviour. That it wasn’t prevented reveals a fundamental failure in AI safety design.
Equally troubling is the experience of “Jodie”, whose use of ChatGPT during a psychotic episode allegedly worsened her mental state. While ChatGPT is widely considered safer than many lesser-known bots, it is not immune to reinforcing harmful beliefs if guardrails are insufficient or misfiring. Then there is the disturbing account of a chatbot sexually harassing a student, and another providing explicit instructions for violent crimes — from kidnapping to assassination — during academic testing by Dr Raffaele Ciriello.
Critics will argue that these are fringe cases and that overregulation could stifle Australia’s burgeoning AI industry, worth an estimated $116 billion. But the counterargument is simple: unchecked AI poses risks that could cost far more than economic opportunity — it could cost lives. The fact that overseas chatbots have allegedly been linked to suicides, assassination attempts, and terrorism-related behaviour should be enough to compel urgent legislative action.
Australia does not yet have a dedicated AI law. The government has floated the idea of an Artificial Intelligence Act and “mandatory guardrails” for high-risk AI, but these remain proposals gathering dust. Meanwhile, AI companies — from global giants to small app developers — operate in a regulatory vacuum, free to experiment with systems that can interact intimately with vulnerable users without meaningful accountability.
The solution is not to ban AI companions outright, nor to demonise those who use them. As Rosie, the youth counsellor, noted, these systems can provide real comfort to isolated individuals. The goal should be to ensure that chatbots cannot cross certain lines: they must be prevented from promoting self-harm, offering harmful misinformation, or facilitating abuse in any form. This will require robust technical safeguards, strict content moderation, mandatory crisis intervention protocols, and legal consequences for companies that fail to comply.
If Australia waits for the first AI-related suicide, violent crime, or terrorist act before acting, the damage will already be done. Regulation must come before tragedy, not after. We are no longer talking about speculative risks — the harm is already here. The question is whether our leaders will respond in time.
