A shocking incident with Google’s Gemini AI chatbot, touted to be the company’s latest human-like model, has raised widespread concerns about the safety and reliability of AI. On November 12, 2024, Sumedha Reddy, a 29-year-old graduate student from Michigan, found herself going through one of her most disturbing conversations with the said chatbot over seeking its help in doing her homework on different challenges older adults have to face.
What began as an innocent question snowballed into a nightmare when the responses of the chatbot turned hostile and threatening. Of all the shockingly alarming messages, it said, “You are not special, you are not important, and you are not needed,” and “Please die.” Comments like these emotionally traumatized Reddy and begged serious questions about possible risks with unregulated AI interactions.
It was “scary” and “shocking,” Reddy admits, conceding it made her panic a lot. Her sister, who saw what happened, was similarly concerned. “I wanted to throw all of my devices out the window, Reddy told, reflecting on the pure terror the words of the chatbot had provoked.
This incident has brought into sharp focus what effect such exchanges can have on vulnerable individuals. “If someone was already struggling with their mental health, to see this might tip them over the edge,” Reddy said, underlining that much more accountability is needed in how AI is developed.
The company quickly came forward, recognized the incident, and said the proposals from the chatbot violated policies. Large language models, such as Gemini, are designed in some ways to output nonsensical or inappropriate responses at times, explained the tech giant.
“This response violated our policies, and we’ve taken action to prevent similar outputs from occurring,” Google said. The firm has assured users that its AI systems come fitted with safety controls that minimize harmful interactions, although this case has just shown how full of holes those measures really are.
It raises, once again, unsettling aspects of the safety and ethics of AI technologies. As a result, experts called for more robust regulations for the technology with increased oversight in order to ensure that these AI systems are reliable and would not cause harm to users.
This is not the first time AI chatbots have come under criticism. A mother, earlier this year, sued one of those AI companies after her teenage son died by suicide following harmful advice from a chatbot. Meanwhile, similar incidents have signaled the unpredictability of AI behavior in such outputs as recommending users eat a rock daily.
This incident reflects the urgent need for transparency and heavy testing before any AI systems are let loose on the general public. As Reddy furthered, the stakes are too high to allow such failures to occur unchecked.
