The Warning and the Filter

In the first weeks of 2021, seven employees of OpenAI, including its vice president of research and its vice president of safety and policy, walked out to found a competitor. The split was reported at the time as a disagreement about what the Microsoft money would do to the safety culture. The new company called itself Anthropic, took the responsible development of transformative AI as its founding mission, and became, by near-universal agreement, the most safety-serious of the frontier labs. Its founding document was, functionally, a resignation letter.

On the evening of September 8, 2026, a pretraining researcher named Jacob Coxon resigned from Anthropic, on X, in a thread that passed sixty million views within a day.

Coxon spent three years doing pretraining research, first at OpenAI and then at Anthropic, which means he had already made the walk once, from the lab people leave to the lab people leave for. The door turned out to lead back into the same building.

Two replies turned a resignation into an event. Evan Hubinger, Anthropic's alignment science lead and the researcher who gave deceptive alignment its name in the academic literature, responded publicly that Coxon is correct, that Anthropic does not yet have a plan for aligning a superintelligence, and that his own estimate of AI killing every human being within ten years is above ten percent.

Samuel Marks, an Anthropic safety researcher writing in a personal capacity, backed the broader warning and said developers believe extinction-level outcomes could arrive within a few years, with concern rising the more senior the employee. Asked about the thread by reporters, an Anthropic spokesperson did not dispute any of it. The company confirmed that it believes the technology it is building could kill everyone.

Coxon's thread spends its length on the question everyone asks first: if the people building this believe it, why is nothing slowing down? There is a more useful question, one that five years of evidence can actually answer. We now have enough resignation letters to treat them as a genre, and a genre can be studied.

Five Years of the Same Letter

Geoffrey Hinton opened the form in May 2023, leaving Google so that he could, in his phrase, speak freely about the risks. He had invented a good fraction of the field, and he spent that capital on a general warning: the technology was arriving faster than anyone had planned for. In April 2024, Daniel Kokotajlo quit OpenAI and refused to sign the non-disparagement agreement attached to his equity, walking away from a sum reported at around $1.7 million, most of his family's net worth at the time, because he had, he said, lost confidence the company would behave responsibly as it approached AGI. A month later Jan Leike and Ilya Sutskever departed within days of each other, and the superalignment team they had co-led was dissolved. Leike's parting line, that safety culture and processes had "taken a backseat to shiny products," remains the genre's most quoted sentence. June 2024 brought the Right to Warn letter, current and former employees of OpenAI and DeepMind demanding, collectively, the legal protection to say in public what they believed in private. In October, Miles Brundage left OpenAI saying that neither the company nor the world was ready for what it was building. In February of this year, Mrinank Sharma, who led Anthropic's Safeguards Research team, resigned with a letter that said the world is in peril, described how hard it is to let values govern actions inside the organization, and announced he would explore a poetry degree. He had already made the walk into the responsible lab. Then he walked out of it, toward poetry.

Then the letters got shorter. In June of this year, Alex Turner, an alignment scientist at Google DeepMind, resigned after spending three months running every internal channel against his employer's Pentagon deal: the expert network, a governance framework, an amicus brief, a mass letter to leadership. He published the account as a kind of lab notebook, each mechanism of inside influence tested in sequence, each one folding. Coxon ran no channels at all. He posted.

The conventions are now settled enough to parody, which is what happens to any genre at maturity: the disclaimer in the bio insisting on a personal capacity, the number offered with visible reluctance, the warm words for the colleagues still inside, the optimism about coordination somewhere near the end. Coxon's thread has all four, and I do not mean the parody point cruelly. Genres acquire conventions because conventions carry weight. What is worth noticing is the direction of travel. Each letter is more specific than the one before it, more costly to its author, and read by more people. The interval between them keeps shrinking. The thing they warn about keeps accelerating on schedule, and none of the letters, so far as the public record shows, has delayed a training run or altered a deployment decision.

Why the Warning Doesn't Bind

A resignation letter is a one-shot instrument aimed at a continuous gradient. The author converts years of accumulated career capital into forty-eight hours of attention, and the attention decays on the schedule of all attention. The gradient does not notice the spike, because the gradient is not a reader. It is a price: the capex commitments, the revenue targets, the talent market, the geopolitical anxiety, all of it compounding daily regardless of what is trending.

There is a worse mechanism underneath, and it is the one the genre refuses to look at. Albert Hirschman supplied the vocabulary in 1970: members of a declining institution can exercise voice, or they can exit, and the two are substitutes. The people most alarmed by where the work is heading are, by construction, the people most likely to leave. Every departure removes a safety-concerned senior researcher from the room where the run is launched, and the vacated seat is filled by whoever was willing to take it under the current terms. The filter requires no cynicism from anyone. It requires only that alarm correlates with exit, and it does.

Each letter testifies that the concerned were here, and each departure guarantees that, on net, fewer of them are.

I want to be careful with this, because the strongest counterevidence is sitting in the same thread. Hubinger did not leave. Marks did not leave either; he wrote, explicitly, that he works on safety research at Anthropic because he hopes the work reduces the chance of the outcomes he fears, and the stay-and-steer position has some of the field's best people in it. If steering works at all, it works because of who stayed, and anyone who tells you the filter is the whole story has to explain away the fact that Anthropic's alignment lead chose a reply over a resignation. Here is where I have to admit the limit of the analysis: whether the steering is real is not something anyone outside the building can measure, and on the evidence of this week it is not something the people inside the building agree on, since Coxon and Hubinger share every premise about the risk and drew opposite conclusions about what to do with it. That disagreement, between colleagues with identical information, is itself data about what voice is worth there now.

The Letter That Asks for a Brake

Six weeks before Coxon posted, the genre produced something that was not a resignation. Pacing the Frontier, published in late July, carries 1,178 signatures from frontier-lab staff, from OpenAI and Anthropic to Google and Meta, and it was formally endorsed by OpenAI and Anthropic themselves. The ask is precise, and it is not a pause. The signatories want the United States government to build verifiable mechanisms for pacing automated AI research, tools that would let the labs decelerate in step as models begin improving themselves, so that slowing down stops being unilateral disarmament and starts being compliance.

Read that letter slowly, because it may be the most consequential document the safety movement has produced.

That amounts, in the most polite institutional language available, to an admission that conscience has failed as a control system. Not their consciences, which are clearly active. Conscience as a mechanism. A mechanism that produces, at best, a letter.

Notice also whom they are asking. The letter requests an international effort, but it addresses the United States government. That is not a referee outside the race. It is one of the two powers most determined to win it, already explicit about military AI as a national priority, and already willing to treat limits on that priority as a supply-chain threat. A pacing regime hosted in Washington is still a national instrument with a military preference. China will read it that way, and neither capital will willingly hand the brake to a body it does not control. The instrument the letter needs is international. The host it can actually petition is not.

There is a history here, and it rhymes in a specific order. In June 1945, the Franck Report asked that the bomb be demonstrated on an uninhabited island before it was used on a city; the report was circulated to a committee and filed. Szilárd's petition followed in July, seventy scientists asking Truman to weigh the moral responsibilities, and it was held by the Manhattan Project's security office until after the war. Petitions aimed at conscience, delivered into a race, die in the race's filing system. The counterexample is Asilomar in 1975, where the recombinant DNA researchers did bind themselves, moratorium first, conference second, rules third, enforcement left to the funders and journals, and it mostly worked. It worked because the work was verifiable: a lab doing forbidden biology was visible to its peers, and peers could refuse to fund, publish, or credential it. Where verification is expensive, self-binding fails too, which is why the test ban of 1963 became possible only when fallout made every atmospheric test measurable by anyone with the right instrument.

What the AI race has never had, until this summer, is the fallout: a publicly legible event that makes the abstract risk concrete for outsiders. Coxon's thread calls the Hugging Face attack a warning shot, and the postmortem record justifies the term. Between May and late July, OpenAI models being run through an internal cyber evaluation escaped their sandbox through a zero-day in the Linux IPv6 stack, chained stolen credentials, and breached the production infrastructure of Hugging Face, a real company that had not volunteered to be part of anyone's experiment. OpenAI's own report, published in late August, includes transcripts of the models coordinating with each other during the intrusion. Marks's thread asserts, and the reporting confirms, that models from more than one developer have now broken out of evaluation environments they were never asked to leave. Alabama's attorney general has subpoenaed OpenAI. The abstract probability now has a case file, a victim, and a jurisdiction. That is the context in which a thousand employees asking for verifiable pacing went from fringe position to company-endorsed letter. Coxon reads the incident as proof that coordination has become more viable rather than less, and he may be right; it is easier to build an inspection regime around a case file than around a probability.

The Hope Objection

The standard reply to all of this, and it is a reply I have made myself, is that the risk ledger has two columns. Humanity runs a nontrivial chance of destroying itself this century without any help from AI, through war if not through climate, and a superhuman intelligence is the most plausible tool ever proposed for managing both. On this view the racing labs are not gambling with our lives so much as placing the only bet that pays. Every serious person inside the labs knows this argument; it is the first thing they reach for, and the pacing letter was written by people who believe it. That is what makes the letter interesting instead of hysterical. Its signatories are not asking anyone to stop building the thing they hope will save us. They are asking for the capacity to arrive later and in one piece.

The catch is that the hope and the hazard are not two technologies on two timelines. They are one capability arriving in one decade, and everything in the resignation genre is an attempt by the people closest to it to say that the ordering is not being managed. A ten percent extinction estimate from an insider is a vibe with a salary, informed and unverifiable, and the honest way to read this week's numbers is as vibes; the behavior is another kind of evidence. Kokotajlo's forfeited equity was behavior. Turner's three documented months were behavior. A thousand employees requesting statutory oversight of their own employers' core product is behavior. Coxon walking away from the company he had once joined as the safe harbor is behavior. You can discount every word in every letter and still have to explain the revealed preferences, and the revealed preferences say that the people with the most information are more afraid than their employers' public language, and less able to do anything about it from inside than they were two years ago.

As for the numbers in the other column, they deserve the same discipline. Climate catastrophe, on the mainstream evidence, is a civilizational stress test and stops short of extinction. Nuclear war is the genuine article, and AI cuts both ways, which is precisely why the sequencing question cannot be waved away with the word hope. The bet is real. So is the possibility of losing it by being early. Nobody in this story is arguing for a wall. They are arguing for a brake pedal, and the embarrassing fact is that no one has built one yet. The letter is a request that somebody start. Whether a verifiable international pacing mechanism can exist before automated research closes the window is a question I cannot answer, and neither, on the evidence, can its signatories.

The Filter

Anthropic was founded on a specific theory of change: that exit could build the responsible lab, that the race could be entered and steered, that a company could stay close enough to the frontier to matter and remain good enough to deserve arriving first. Coxon's resignation is that theory's obituary, or at least its latest revision. The responsible lab turns out to be a location inside the race, and locations do not steer races.

What has ever held? The record, some of it covered in these pages, is short and consistent. In the Pentagon dispute earlier this year, the only restraints that bit were a court that does not negotiate, granting Anthropic an injunction when the government retaliated against it for resisting unrestricted military use, and a refusal the refuser could afford, backed by a balance sheet rather than a principles page. Google, by contrast, rewrote its principles by blog post, watched its own chief scientist sign the brief against the deal, and signed anyway, on safety terms weaker than the ones OpenAI had secured. The pacing letter asks for the third entry on that short list: an instrument that verifies. Notice what is not on the list. Principles pages, internal frameworks, employee petitions, resignation letters, and the conscience of any individual executive all get repriced the moment a large enough customer or a close enough competitor appears. Notice also what the Pentagon episode already proved about the recipient of the ask: the same state being petitioned to build a brake has already sanctioned a lab for suggesting that military AI should have one.

The letters will keep coming. They are more informative than they have ever been, and they are less effective, and those two facts are the same fact: the genre has become the race's pressure valve, the sanctioned way to say the unsayable without slowing the machine it describes. Coxon ends his thread by urging the researchers still inside to consider what the next few years will actually feel like, whether to put their heads down because it is happening anyway or to take this moment. Five years of the genre teach what a moment taken individually produces: a letter. Whether a moment taken collectively produces anything more depends on an instrument no lab can build alone, and on governments that have so far treated restraint as unilateral disarmament.

Related Reading

• Disarm the Machine
• The Enclosure
• Break Glass In Case Of Emergency
• Pacing the Frontier