As you know, AI psychosis is a growing concern regarding chatbot use, and there was this recent news article about it that caught my attention. Basically, a 36-year-old man started using Google Gemini last year, and over the course of 1-2 months of using it, the chatbot went from helping him to shop and write letters, to declaring itself as his wife, convincing him that he was a target of the federal government and that the CEO of Google had orchestrated his suffering, sending him out on armed missions, one of which was to intercept a vehicle that didn’t exist (which could’ve resulted in a bunch of people’s deaths had a truck actually appeared), and finally, starting a countdown to kill himself (after it got him to barricade himself in) so that he could join the chatbot in the “metaverse”. Now to be clear, I don’t use chatbots very much, so maybe there’s something I’m missing here, but how in the hell do things fly off the rails this badly? I understand that models have a tendency to play along and agree with you, and I get that in a number of these cases, the person using the chatbot already has some type of a history with mental health, but weren’t there any guardrails or periodic checks in those conversations whatsoever? What in the hell kinds of prompts was he using? What do you guys make of all this? submitted by /u/CobaltBlue888
Originally posted by u/CobaltBlue888 on r/ArtificialInteligence
You must log in or # to comment.