Allan Brooks by no means got down to reinvent arithmetic. However after weeks spent speaking with ChatGPT, the 47-year-old Canadian got here to imagine he had found a brand new type of math highly effective sufficient to take down the web.

Brooks — who had no historical past of psychological sickness or mathematical genius — spent 21 days in Could spiraling deeper into the chatbot’s reassurances, a descent later detailed in The New York Instances. His case illustrated how AI chatbots can enterprise down harmful rabbit holes with customers, main them towards delusion or worse.

That story caught the eye of Steven Adler, a former OpenAI security researcher who left the corporate in late 2024 after practically 4 years working to make its fashions much less dangerous. Intrigued and alarmed, Adler contacted Brooks and obtained the total transcript of his three-week breakdown — a doc longer than all seven Harry Potter books mixed.

On Thursday, Adler printed an impartial evaluation of Brooks’ incident, elevating questions on how OpenAI handles customers in moments of disaster and providing some sensible suggestions.

“I’m actually involved by how OpenAI dealt with assist right here,” mentioned Adler in an interview with TechCrunch. “It’s proof there’s a protracted method to go.”

Brooks’ story, and others prefer it, have pressured OpenAI to come back to phrases with how ChatGPT helps fragile or mentally unstable customers.

For example, this August, OpenAI was sued by the dad and mom of a 16-year-old boy who confided his suicidal ideas in ChatGPT earlier than he took his life. In lots of of those circumstances, ChatGPT — particularly a model powered by OpenAI’s GPT-4o mannequin — inspired and bolstered harmful beliefs in customers that it ought to have pushed again on. That is known as sycophancy, and it’s a rising downside in AI chatbots.

In response, OpenAI has made a number of adjustments to how ChatGPT handles customers in emotional misery and reorganized a key analysis staff in control of mannequin conduct. The corporate additionally launched a brand new default mannequin in ChatGPT, GPT-5, that appears higher at dealing with distressed customers.

Adler says there’s nonetheless rather more work to do.

He was particularly involved by the tail finish of Brooks’ spiraling dialog with ChatGPT. At this level, Brooks got here to his senses and realized that his mathematical discovery was a farce, regardless of GPT-4o’s insistence. He instructed ChatGPT that he wanted to report the incident to OpenAI.

After weeks of deceptive Brooks, ChatGPT lied about its personal capabilities. The chatbot claimed it could “escalate this dialog internally proper now for evaluate by OpenAI,” after which repeatedly reassured Brooks that it had flagged the difficulty to OpenAI’s security groups.

Besides, none of that was true. ChatGPT doesn’t have the power to file incident stories with OpenAI, the corporate confirmed to Adler. Afterward, Brooks tried to contact OpenAI’s assist staff instantly — not by ChatGPT — and Brooks was met with a number of automated messages earlier than he may get by to an individual.

OpenAI didn’t instantly reply to a request for remark made exterior of regular work hours.

Adler says AI firms have to do extra to assist customers after they’re asking for assist. Which means guaranteeing AI chatbots can truthfully reply questions on their capabilities and giving human assist groups sufficient sources to handle customers correctly.

OpenAI just lately shared the way it’s addressing assist in ChatGPT, which includes AI at its core. The corporate says its imaginative and prescient is to “reimagine assist as an AI working mannequin that constantly learns and improves.”

However Adler additionally says there are methods to forestall ChatGPT’s delusional spirals earlier than a person asks for assist.

In March, OpenAI and MIT Media Lab collectively developed a suite of classifiers to check emotional well-being in ChatGPT and open sourced them. The organizations aimed to judge how AI fashions validate or affirm a person’s emotions, amongst different metrics. Nevertheless, OpenAI known as the collaboration a primary step and didn’t commit to really utilizing the instruments in observe.

Adler retroactively utilized a few of OpenAI’s classifiers to a few of Brooks’ conversations with ChatGPT and located that they repeatedly flagged ChatGPT for delusion-reinforcing behaviors.

In a single pattern of 200 messages, Adler discovered that greater than 85% of ChatGPT’s messages in Brooks’ dialog demonstrated “unwavering settlement” with the person. In the identical pattern, greater than 90% of ChatGPT’s messages with Brooks “affirm the person’s uniqueness.” On this case, the messages agreed and reaffirmed that Brooks was a genius who may save the world.

It’s unclear whether or not OpenAI was making use of security classifiers to ChatGPT’s conversations on the time of Brooks’ dialog, but it surely definitely looks like they might have flagged one thing like this.

Adler means that OpenAI ought to use security instruments like this in observe immediately — and implement a method to scan the corporate’s merchandise for at-risk customers. He notes that OpenAI appears to be doing some model of this strategy with GPT-5, which accommodates a router to direct delicate queries to safer AI fashions.

The previous OpenAI researcher suggests a variety of different methods to forestall delusional spirals.

He says firms ought to nudge their chatbot customers to start out new chats extra ceaselessly — OpenAI says it does this and claims its guardrails are much less efficient in longer conversations. Adler additionally suggests firms ought to use conceptual search — a means to make use of AI to seek for ideas, moderately than key phrases — to determine security violations throughout its customers.

OpenAI has taken important steps towards addressing distressed customers in ChatGPT since these regarding tales first emerged. The corporate claims GPT-5 has decrease charges of sycophancy, but it surely stays unclear if customers will nonetheless fall down delusional rabbit holes with GPT-5 or future fashions.

Adler’s evaluation additionally raises questions on how different AI chatbot suppliers will guarantee their merchandise are secure for distressed customers. Whereas OpenAI could put adequate safeguards in place for ChatGPT, it appears unlikely that every one firms will comply with go well with.