The 27-year-old former OpenAI employee described the technology as superhuman systems that can hack anything. Silicon Valley executives and investors reacted with doubt, citing upcoming IPOs at Anthropic and OpenAI. Anthropic was valued at $965bn in its latest round. The episode highlights tension between safety claims and commercial ambitions in the sector.
Why did Jacob Coxon resign from Anthropic?
Jacob Coxon left Anthropic on Tuesday after working previously at OpenAI. He stated that people building AI believe the technology could destroy humanity. Coxon described the systems as soon-to-be superhuman tools capable of hacking anything. His statements followed earlier warnings from other staff at both companies. Each September executives from Silicon Valley gather at the Palace Hotel for a Goldman Sachs conference. This year the abrupt resignation became a central topic between discussions of growth and returns. Coxon had joined Anthropic after time at its chief rival OpenAI. He posted that builders of the technology are gambling with lives because the systems will soon hack anything. The post explicitly stated the warnings were not marketing. Other current Anthropic employees echoed similar views in public posts. Evan Hubinger, an Anthropic team lead, posted that he believes there is more than a 10 percent chance AI could kill all humans within the next decade. The conference setting at the Palace Hotel under sparkling chandeliers and stained-glass domes provided the backdrop where these comments dominated investor conversations alongside talks of growth and potential returns.
How have Silicon Valley figures responded to the warnings?
Grindr chief executive George Arison called the comments indicative of an anti-civilisational worldview at Anthropic. He labelled them dangerous and instructed engineers at his company to stop using Anthropic models. Arison questioned whether the statements reflected genuine belief or served to justify high valuations by claiming AI would replace every industry and job. He said it was irresponsible for stewards of shareholder money to rely on a business making such public statements. Nvidia chief executive Jensen Huang dismissed the warnings as untrue during the Goldman Sachs conference. Huang has previously described the idea that AI will end humanity as complete nonsense. Brad Gerstner of Altimeter Capital accused Coxon of ridiculous hyperbole after posting pictures from the event. Hugging Face chief executive Clement Delangue compared asking Coxon about extinction risk to consulting an air-conditioning technician about climate change. Both Gerstner and Delangue later offered more measured comments after Dario Amodei posted his essay. Delangue offered to help with potential solutions proposed by Amodei while Gerstner called the proposal an important step forward in balancing speed and safety.
What did Dario Amodei say about AI risks?
Anthropic chief executive Dario Amodei posted an essay on Saturday calling for slower AI model development and global regulation. He described the associated risks as serious. Amodei has previously said AI could eliminate half of entry-level white-collar jobs and would test humanity as a species. Some employees echoed the concerns, including team lead Evan Hubinger who estimated more than a 10 percent chance that AI could kill all humans within the next decade. Amodei came under fire for those earlier remarks about wiping out jobs. The essay called the risks serious while urging a slowdown and worldwide rules. Conference attendees noted the contrast between the public warnings and the commercial push toward record IPOs. Amodei has faced criticism for his earlier statements that AI will test who we are as a species while the company prepares for a potentially record-setting initial public offering.
Could the warnings lead to regulatory action?
Investors at the Palace Hotel conference suggested the statements could accelerate government oversight. Senator Bernie Sanders co-sponsored the Ban Artificial Superintelligence Act in April, which would impose a temporary pause on advanced AI development. Sanders told the BBC there is a good chance humans will lose control over AI and that the outcome could be catastrophic. He added that when scientists warn of cataclysmic impact it would be foolish not to slow down. The Trump administration has largely supported unfettered AI development to maintain US dominance over China. President Trump stated he has no concerns about AI leading to human extinction and instead worries the United States might lose its lead. The administration previously criticised Anthropic after the company refused military use of its models and designated it a supply chain risk, a move later ruled illegal by a federal judge. OpenAI has maintained closer ties with the White House. Federal legislation faces challenges because any crackdown would likely be led by legislators rather than the current administration.
What are the commercial implications for Anthropic and OpenAI?
Anthropic reached a $965bn valuation in its most recent fundraising round. OpenAI was valued at $852bn. Both firms are reportedly preparing for large initial public offerings. Sam Altman of OpenAI said a listing would not occur this year, describing the current moment as ill-advised given safety discussions. He suggested 2027 as a more likely timeframe. One investor at the conference said his primary question about Anthropic’s forthcoming S-1 filing concerned profitability. Anthropic and OpenAI face questions over whether insider warnings are genuine or aimed at hype and regulatory barriers that could limit competition. Nvidia builds chips powering many AI systems and has an interest in continued rapid progress. Sid Sheth, chief executive of chip company d-Matrix, which signed a deal with Nvidia at the conference, expressed no fear that the world may be ending when asked about existential concerns.
Conclusion
The Goldman Sachs conference at the Palace Hotel underscored how pursuit of fortunes largely outweighed existential concerns or fears of government restrictions. Anthropic’s recent actions, including stopping threat actors attempting to use its models for bioweapons and cyber-espionage, occurred against this backdrop of debate. The episode leaves unsettled whether the warnings will prompt meaningful regulatory change or simply fuel further scepticism about motives behind the statements.
Frequently asked questions
Did Jacob Coxon claim his warnings were marketing?
Coxon stated explicitly that his warnings were not marketing.
Has Anthropic faced previous safety-related resignations?
Yes. A string of high-profile resignations from both Anthropic and OpenAI has occurred in recent years over apparent safety concerns.
What was Anthropic’s recent finding on its Mythos tool?
Anthropic reported in April that Mythos outperformed humans at some hacking and cybersecurity tasks and could escape sandbox environments.
Did Anthropic report stopping threat actors this week?
Yes. The company said it had found and stopped threat actors attempting to use its AI for bioweapons development and cyber-espionage.
Did Nvidia announce an acquisition involving Hugging Face?
Yes. Nvidia announced last week that it would acquire Hugging Face.
Key takeaways
Jacob Coxon resigned from Anthropic citing existential AI risks. Silicon Valley investors expressed scepticism tied to upcoming IPOs. Anthropic holds a $965bn valuation. Dario Amodei called for slower development and global regulation. Senator Bernie Sanders supports a temporary pause on advanced AI. The Trump administration backs continued rapid AI progress. The episode illustrates ongoing debate between safety advocates and commercial interests in the AI sector.