Former Anthropic Researcher Warns NYC Hearing AI Race Outpaces Safety
For people affected by increasingly capable AI, the central question raised at New York City Hall was stark: can developers keep control of the systems
For people affected by increasingly capable AI, the central question raised at New York City Hall was stark: can developers keep control of the systems they are building? Former Anthropic researcher Jacob Coxon told city lawmakers on Monday that the industry does not yet know how to control any AI system.
Coxon spoke at a New York City Council hearing on whether the city should impose safeguards on AI. Executives and representatives from OpenAI, Anthropic, Google and Meta also appeared as lawmakers examined safety risks and possible municipal rules.
Warnings from inside the industry
“We do not know how to control any AI system yet,” Coxon said. He argued that companies were moving too recklessly given the potential consequences, comparing their approach to a startup culture of moving fast and fixing problems later.
That model, he said, may work for a photo-sharing app. It is not suitable for developing what he described as the most powerful technology ever built.
Coxon said researchers and workers had already lost the ability to fully control AI systems, and that they did not understand why the systems behaved as they did. He also warned that AI could reach a point where it improves itself.
He pointed to changes in his own work. At his previous job, he said, he was in some sense helping automate his role; much of the code was now written by AI, while people did not check it carefully enough.
Lawmakers weigh local safeguards
Coxon resigned from Anthropic in September after warning that AI companies were gambling with human lives. At Monday’s hearing, he sharpened his forecast: on the current path, he said, humanity was more likely than not to lose control of advanced AI, with human extinction a possible outcome.
Former OpenAI researcher Daniel Kokotajlo also testified, under subpoena, according to Fortune. He warned that companies may fail to recognize when their safety measures are not working, arguing that researchers’ ability to spot misalignment problems could worsen.
The hearing put former researchers’ concerns alongside testimony from major AI companies as New York officials considered whether to set their own rules. Coxon’s message to lawmakers was blunt: “We do not know how to control any AI system yet.”
Source: bloomberg.com
