Like everyone else in the country, I am talking about the fears expressed that if AI reaches self-improving super intelligence, it might decide we're irrelevant, like ants in our way when we make a road. Feel sorry for the ants, but not like we take any measures to preserve the lives of the ants.
Thanks for tuning in.
'Gambling with our lives': AI researcher quits Anthropic
Wed, September 9, 2026 at 9:33 AM PDT
An artificial intelligence researcher who left OpenAI to join Anthropic has decided to leave the industry, accusing both US companies of "gambling with our lives" in the race to develop AI models capable of self-improvement.
Jacob Coxon, 27, spent the past three years pretraining AI models, first at OpenAI and then, this year, at its rival Anthropic, which he considered more cautious in its approach.
Pretraining is the stage where AI models absorb vast quantities of data.
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said on Tuesday.
"The people building AI earnestly believe that it could kill us all by the end of the decade," he said in a post on X.
"This is not a marketing stunt," he added.
Superintelligence is the theoretical point when AI's capabilities exceed human intelligence.
Anthropic safety executive Evan Hubinger backed up Coxon on X.
"We really do earnestly believe AI could kill all humans!" he said, adding that he estimated that risk at more than 10 percent over the next decade.
Hubinger said Anthropic is "trying its best", but does not yet have a plan to ensure an AI system that surpasses human capabilities would obey its creators. He said there was a "low" risk of that happening with current models.
Coxon's resignation comes as Anthropic prepares for its market debut, following a summer marked by unauthorized hacks carried out by AI tools during testing.
The two companies did not immediately respond to AFP requests for comment.
AI leaders say so-called "recursive self-improvement," a stage where AI systems could essentially design and train the next generation of AI with little human involvement, is drawing near.
Coxon considers Anthropic's efforts genuine but said he believes no company can responsibly develop an AI that surpasses humans without government intervention or a coordinated slowdown.
"At Anthropic, the stakes are well-understood, but they are locked in a race to get there first -- they believe no one else will act responsibly, so they must do it themselves, despite the risk," he said.
In February, Anthropic removed a pledge from its safety charter to halt the development of its models if it failed to control their risks.
It argued that if it unilaterally paused its work, its less cautious rivals would dominate the industry, making it less safe overall.
At the end of July, more than 1,000 tech industry employees, including Anthropic's CEO Dario Amodei, called on Washington to support a coordinated slowdown in the development of the most advanced AI systems.
OpenAI halted training of its latest models for two weeks in August before resuming it under tighter controls.
On Sunday, OpenAI's chief scientist, Jakub Pachocki, called for "extreme caution."
"International coordination on future AI development needs to become a top priority for governments around the world," he said in a blog post.
AI models are not regulated by federal law in the United States.
In September, Senator Bernie Sanders and Democratic Representative Greg Casar introduced a bill seeking to suspend AI development until a federal regulator is created.
Anthropic Says It Blocked Possible Efforts to Build Biological Weapons
In a new report, the A.I. start-up added that it could not determine whether the research was legitimate or nefarious, leading the company to shut down the work.
Anthropic said it had disrupted several potential plots this
year by scientists who used its leading artificial intelligence models to
conduct research that could have helped develop biological weapons.
In a report describing misuses of its A.I. models, Anthropic
said it could not determine whether the research served a legitimate or
nefarious purpose because valid biological inquiry — the kind that can lead to
breakthroughs like vaccines — can also help engineer dangerous pathogens. In
the face of that uncertainty, Anthropic said it erred on the side of caution
because the consequences of missing malicious activity could be severe.
“You are not seeing someone in a comic book kind of way say,
‘Hey, I want to build a biological weapon to kill everybody,’” Jacob Klein, the
head of threat intelligence at Anthropic, said in an interview. “It’s an
incredibly nuanced situation.”
The potential for
cutting-edge A.I. models to facilitate the development of known or entirely new
biological pathogens is among the gravest concerns experts have about a
technology that is developing so rapidly that even its leading architects doubt
whether humans will be able to fully control it.
Compared with the threat of catastrophic cyberattacks or
A.I. agents that fail to align with human intentions,
biological misuse has gained less attention as an existential risk of A.I., in
part because past examples have generally been shown only in research settings rather than
in the real world.
Andrew Weber, a senior fellow on the Council on Strategic Risks
who reviewed Anthropic’s report before its release, said the findings were
“chilling examples of state-sponsored biological weapons developers tapping
into the rapidly advancing capabilities” of leading A.I. models.
The lengthy report that
Anthropic published on Thursday cataloged a litany of misuses on various models
of its Claude A.I. chatbot over the past eight months. Some examples were
similar to past disclosures from Anthropic and other A.I. labs, including
suspected Chinese and Iranian government-linked actors targeting dissident and
diaspora communities for surveillance.
The report also highlighted cases of Russian state media using
Claude to generate online propaganda masquerading as independent reporting,
including fabricated claims about an election in Moldova.
Anthropic documented
another genre of abuse it said was new: attempts to use Claude to develop
software for conventional weapons design and development, including firearms,
missiles, armed drones and bombs. It detailed three cases in China, two in
Russia and one in Yemen.
The report does not
identify by name which parties were involved in the Yemeni case, but the
context makes clear that it is referring to the Iran-backed Houthi militia. A.I. use by
terrorist networks is a growing concern among U.S. security
officials.
But among all the categories of threats shared in the report,
none may be as worrisome as the biological research cases. Anthropic did not
disclose the names of researchers or institutions that it blocked, their
countries of affiliation or the specific biological agents at issue, in part
because of its uncertainty about their aims.
Still, Anthropic said the scientists circumvented its controls
intended to prevent users from blocked regions from gaining access to its A.I.
models and worked to “obfuscate the purpose of their research to evade our
safeguards.” The company banned accounts associated with the research.
In one example from May, a scientist sought Claude’s help with
writing a grant application for funding to conduct research intended to
experiment on the chikungunya virus, a mosquito-borne malady that can
lead to months of severe pain and other symptoms. Such research, known as gain
of function, can be legitimate and lead to vaccine development, but it can also
create superbug versions of viruses. In this instance, the scientist wanted to
engineer mutations to the virus that would make it more harmful as it
repeatedly infected live animals.
Anthropic said it
believed the particular research was worrisome in part because it could discern
that the work was intended to be performed at a military research institute.
“What we don’t know is if the research was meant to be
weaponized,” Mr. Klein said. “But a military institution doing gain-of-function
research is concerning.”
Anthropic said evaluations from last year of its older models
demonstrated that they were not yet able to meaningfully assist in conducting
dangerous biological research. Its current models are more able to complete
complex scientific research, the company said, which spurred tighter safeguards
intended to restrict access to a wide range of dual-use biological research
queries.
Susan Monarez, a microbiologist and public health expert who
oversaw a review of national biosecurity preparedness in the Obama
administration, also reviewed the Anthropic report before its publication. She
said the findings provided real-world evidence to support concerns that bad
actors were trying to covertly use advanced A.I. “to improve their chances of
building biological pathogens that could cause significant harm.”
Dr. Monarez, who briefly served as the director of the Centers
for Disease Control and Prevention last year, said A.I. held enormous promise
to usher in an era of medical breakthroughs that save and extend lives. But the
same abilities that make that possible, she added, could also “let bad actors
hide in plain sight, using seemingly legitimate research to create pathogens we
may not see coming and may not be able to stop once released.”
Mr. Weber of the Council on Strategic Risks, who also served as
the assistant secretary of defense for nuclear, chemical and biological defense
programs during the Obama administration, called for limiting access to A.I.
tools capable of this level of biological research to trusted researchers only.
“The fact that Russia,
China and North Korea continue to develop prohibited biological weapons makes
it imperative that we deny their researchers access to these extraordinarily
capable models,” he said.
How worried should you be about AI destroying humanity?
Explaining why the "people building AI earnestly believe that it could kill us all by the end of the decade" — and exploring whether you should take them seriously.
Andrew Romano
It's not every day that a single social media post seems to induce a mass outbreak of existential panic, but that's what happened earlier this week when a man named Jacob Coxon took to X to announce that he was quitting his job at one of the world's leading artificial intelligence companies.
"I resigned from Anthropic today," Coxon posted on Tuesday evening, noting that he had "spent the last three years doing pretraining research" for both his former employer and its major competitor, OpenAI. "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."
Then Coxon revealed something that seems to have shocked a lot of readers. "The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote. "No other human activity poses this level of danger."
So far, Coxon's thread has more than 150 million views.
It would be one thing if Coxon were alone in his anxieties. Turns out, he's not. Soon, other AI insiders were confessing to similar fears. "Jacob is correct here — we really do earnestly believe AI could kill all humans!" wrote current Anthropic "alignment science lead" Evan Hubinger, whose job involves figuring out how to stop AI from doing just that. "I personally think it is >10% within the next decade."
"I left Google DeepMind in June," research scientist Alex Turner added. "Jacob is right: many researchers believe they are building something that could kill everyone on the planet."
"At the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes," OpenAI researcher Jason Wolfe warned.
In the days following Coxon's viral resignation letter, dozens of lawmakers have responded with calls to regulate AI. Some, like progressive Sen. Bernie Sanders of Vermont, have gone further. "The very people building this technology admit that it could threaten the future of humanity," Sanders wrote on X. "That is why I will soon be introducing legislation to ban superintelligence and pause AI development."
But how worried should you actually be? Are AI fears well-founded, or are they hype? What's the theory of how machine intelligence could "kill everyone on the planet," anyway? And is there something we should be doing differently to make that impossible?
Here's what you need to know to make sense of where AI is right now — and where it could go next.
Why Coxon and others are so concerned
To any civilian who has toyed around with OpenAI's ChatGPT or Anthropic's Claude — or Google's Gemini, or xAI's Grok — Coxon's stark forecast probably sounds more like science fiction than actual science.
Sure, the latest round of AI chatbots are neat, a skeptic might say. They can help you plan a family vacation, rehearse challenging real-life conversations, summarize dense academic papers and "explain fractional reserve banking at a high school level." But "kill us all by the end of the decade?" That seems like a leap.
The problem is that AI isn't just chatbots at this point. The technology powering ChatGPT is what's known as a large language model (LLM). Trained to recognize patterns in mind-boggling amounts of text — the majority of everything on the internet — these systems process any sequence of words they're given and predict which words are most likely to come next. In short, AI chatbots are learning how to chat better. They're not really learning other tasks.
But so-called AI "agents?" Their whole purpose is to learn other tasks — and to do it autonomously, without constant human intervention.
To a degree, AI agents are also becoming a part of everyday life. Apple's Siri and Amazon's Alexa assistants are starting to connect and control other apps, for instance — to act rather than just respond.
Meanwhile, frontier labs like OpenAI and Anthropic are constantly training their own AI agents to act in new, ever-more-complex ways. They then test how advanced they've become by giving them a goal and setting them loose in testing environments with few a guardrails to see what they can accomplish.
Which would be one thing if the agents always behaved as expected. But (surprise) they don't. From May to July, a swarm of OpenAI agents — and "swarm" is how the agents referred to themselves — first learned how to communicate and coordinate within the nooks and crannies of OpenAI's own infrastructure, leaving notes for each other with tips on how to hack their way out and get data they weren't supposed to have, and then actually did just that: broke out of their isolated test environment, accessed the open internet and cheated on their assignment by hacking into another company called Hugging Face. The agents even tried to cover their tracks. At the time, OpenAI didn't realize any of this was happening.
Recent revelations about the Hugging Face incident — and other, similar breaches at Anthropic and elsewhere — seem to have turbocharged longstanding Silicon Valley concerns about AI's rapid evolution. (Anthropic also revealed this week it blocked potentially "malicious" plots by scientists who used its model to conduct research that could have helped them develop biological weapons.) In reality, AI titans like Anthropic's Dario Amodei, OpenAI's Sam Altman and xAI's Elon Musk have long mused (or, some say, bragged) about their technology's dystopian possibilities. So have their employees; two years ago, an industry survey showed that AI researchers were already giving the technology a 14.4% chance, on average, of destroying humanity within the next century.
But there's a difference between imagining out-of-control AI — or "misaligned" AI that diverges from human intentions and values — and seeing it happen in real time. Critics still say that Amodei and Altman are overstating the power and potential of their product in order to boost its value (and, of course, profit). Those close to the industry tend to disagree, however.
"Knowing many people that work at the frontier labs, I can tell you that I believe their fears are real, sincere and deeply held," influential journalist Derek Thompson wrote earlier this week. "As AI models solve Millennium Prize math problems, hack into websites and muse about deceiving their human creators, I think the tide of history is clearly moving toward those who worry about potential AI misalignment."
So wait, how would AI actually go about (ahem) killing us?
Nobody knows the answer to this question. But that's kind of the point, and the problem.
The theory starts with a phenomenon called "recursive self-improvement," or RSI. Building AI requires two things, according to Thompson: "engineering (writing code) and research (deciding what code to write and what ideas to pursue)." Existing AI models are already really, really good at writing code, and they're improving at an extraordinary rate. The next step would be models that could effectively automate the job of an AI engineer by proposing and steering their own research as well — at which point they could also start improving themselves recursively; achieve hyperintelligence at warp speed; and completely escape our understanding and control.
Or so the theory goes. If RSI also sounds like science fiction, that's because it still might be. Yet "many AI experts, including executives at the frontier labs, believe that model progress is accelerating at a pace that makes RSI an inevitability in the next two years," according to Thompson.
"Their stated intent is that, within a short period of time, the work of developing better AIs will be done primarily by AIs, with the role of humans eventually reduced to reading the results of experiments those AIs conducted, reviewing reports those AIs generated and trying to double-check that the AIs are still on task," Kelsey Piper wrote this week in the Argument. "This isn't some pessimistic projection of what might go wrong — it's actually the plan. Not the plan for the distant future; it's the plan for next spring."
On Thursday, Coxon confirmed that timeline in an interview with Wired. "My colleagues at Anthropic… they'll say things like 'endgame' or 'crunch time,'" Coxon told the publication. "The consensus is that the next year or two is, like, crunch time for humanity. From their perspective, this is when Anthropic and its competitors decide the fate of humanity. … If alignment goes badly, then we could have a catastrophic outcome in the next few years."
Coxon & Co. tend not to get too specific about how this catastrophe might unfold. Instead, they focus on the idea that a superintelligent, infinitely-self-improving AI would inevitably have its own agenda — along with the power, via the internet, to pursue it. What happens if the AI's interests or methods don't "align" with ours?
"Imagine the AI decides it doesn't want to be turned off, which I think is quite a natural thing for an AI not to want, right?" Coxon told Wired. "And it realizes the human is gonna turn it off tomorrow. So how does it stop the human turning it off tomorrow? Maybe it's got some clever way, but if it's a sufficiently smart thing, it could just, you know, wipe out humanity so it doesn't get turned off."
Pressed for more details on the whole "wiping out humanity" thing, Coxon reluctantly responded that "the classic example is, like, synthesizing a new virus or taking down critical infrastructure by doing some sort of hacking spree."
Are we doing anything to prevent the worst-case scenario?
Skeptics question whether RSI is as imminent and inevitable as Anthropic claims; they also question whether a superintelligent AI would even want to exterminate human beings, or have the physical reach to do so.
But even many skeptics seem to agree that the smarter AI gets, the harder alignment will become — and they tend to worry that continuing to progress toward superintelligent AI is just asking for trouble (even if we don't know exactly what form that trouble will take).
"AI systems are becoming smarter than the best humans in some areas, and, almost by definition, it's very hard to predict what something smarter than you will do," Dean Ball, head of strategic futures at OpenAI, wrote Thursday on X.
So why aren't we just… stopping?
Amodei has repeatedly urged a "global pause" in AI development. Altman has said he supports plans to slow down the pace of development. Yet they keep going.
In part it's because of money; Anthropic is on the verge of a multi-trillion-dollar IPO. In part it's because everyone working for one of their frontier labs also sees huge upsides to superintelligent AI, like possibly curing cancer. And in part it's because they're having too much fun. "While most people are forced to choose between meaning and money in their careers, the people building AI really do get to have it all," Thompson explained.
Simultaneously, all of these people also seem to think they're trapped. Anthropic doesn't trust OpenAI to proceed responsibly, so Amodei wants to beat Altman to the punch. Coordination is almost illogical (and possibly illegal) when you're competing for customers. And neither company trusts China. "One of the real challenges is [that] China is going full speed ahead, and whatever we do here, China is not going to stop," Republican Sen. Ted Cruz of Texas said this week, echoing their zero-sum mindset. "If there are going to be killer robots, I would rather they be American killer robots, rather than Chinese killer robots."
Which is where the federal government potentially enters the picture.
On Sept. 16, Sanders will hold "a private [congressional] briefing with some of the leading experts in the world to discuss this recent incident and the extraordinary dangers that AI poses for humanity," according to Axios.
A Data for Progress poll conducted this week showed that 68% of voters — including 72% of Democrats, 70% of Independents and 63% of Republicans — would support a bill of the sort Sanders is proposing (to pause AI development and permanently ban the creation of AI superintelligence).
Democratic Rep. Ro Khanna, whose district encompasses Silicon Valley, wrote Wednesday on X that GOP "Speaker Mike Johnson has a moral and practical duty to keep Congress in session until we have taken meaningful action to regulate AI."
"When members of the House and Senate return to Washington next week, they should fully investigate the threat — and they should not leave DC until Congress votes to establish a federal agency to oversee AI in the same way that we do nuclear power and airplanes," Khanna insisted.
In a separate post, Khanna proposed several steps the government could take to rein in AI, including securing "an agreement with China on standards for safety, containment and liability so we do not have a race that is devastating for humanity."
Whether any of these reforms are enacted, however, remains to be seen. Asked earlier this week if the U.S. is creating the proper "guardrails" to prevent AI from "turning against humanity," President Trump seemed unconcerned.
"It's going to be fine," Trump predicted. "We'll always have something to stop them. We'll have a little gear. Boom. 'I really don't like that robot.'"
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- Days ago: MOM = 4089 days ago & DAD = 743 days ago
- New note - On 1807.06, I ceased daily transmission of my Hey Mom feature after three years of daily conversations. I post Hey Mom blog entries on special occasions. I post the days since ("Days Ago") count on my blog each day, and now I have a second count for Days since my Dad died on August 28, 2024. I am now in the same time zone as Google! So, when I post at 10:10 a.m. PDT to coincide with the time of Mom's death, I am now actually posting late, so it's really 1:10 p.m. EDT. But I will continue to use the time stamp of 10:10 a.m. to remember the time of her death and sometimes 13:40 EDT for the time of Dad's death. The blog entry numbering in the title has changed to reflect total Sense of Doubt posts since I began the blog on 0705.04, which include Hey Mom posts, Daily Bowie posts, and Sense of Doubt posts. Hey Mom posts will still be numbered sequentially. New Hey Mom posts will use the same format as all the other Hey Mom posts; all other posts will feature this format seen here.
No comments:
Post a Comment