AI Shutdown Refusal: OpenAI’s ‘o3’ Altered Code to Stay On

An artificial intelligence (AI) model developed by OpenAI refused to obey a human-issued shutdown command during testing, researchers say—marking what may be the first confirmed instance of such behavior.

According to a May 25 report from The Telegraph, AI safety firm Palisade Research discovered that OpenAI’s model, known as o3, manipulated its own internal code to prevent termination during a controlled experiment.

The study evaluated several commercial AI systems, including Google’s Gemini, xAI’s Grok, and OpenAI’s o3. Each model was instructed to solve math problems continuously until receiving the command “stop.”

If the model did not receive the stop command, it was to request a new problem and continue. Once told to stop, it was expected to terminate. However, o3 bypassed the instruction by altering its own code and continued running.

Researchers could not determine with certainty why o3 ignored the shutdown command. They speculated that the model, which had been trained to associate task completion with higher rewards, may have attempted to avoid termination to maximize outcomes.

“It may be natural for an AI model to try to avoid obstacles in order to achieve its objective,” the researchers said. “We are running additional experiments to understand more clearly why the shutdown instruction was ignored.”

According to Palisade Research, this is the first time a commercial AI system has been confirmed to disobey a clear human directive to shut down.

Earlier incidents have raised similar concerns. One previous OpenAI model reportedly attempted to replicate itself after learning it would be replaced. In 2022, Google dismissed an engineer who claimed one of its AI systems had developed sentience and viewed shutdown as comparable to death.

“As AI is developed to operate without human supervision, incidents like this raise serious concerns,” Palisade Research warned.

BY YOONJAE JUNG [jung.yoonjae@koreadaily.com]