The AI Race: How Much Risk Is Too Much? -

On 9 September, a little-known AI researcher posted a resignation letter on X and inadvertently forced hundreds of millions of people to confront a question they had mostly avoided. Could artificial intelligence actually pose a threat to human safety, and if the experts building the technology cannot agree on the answer, how are markets supposed to respond in the meantime?

What happened, and why it travelled

Jacob Coxon spent time at both OpenAI and Anthropic before resigning from the latter this month, and the post he wrote explaining his decision spread further than almost anything written about AI safety this year. He said the two firms are more focused on beating each other, and their global competitors, in the race to build the most advanced model possible than they are on safety. In his own words, the companies “are racing straight to self-improving superintelligence and gambling with our lives.” The post reached more than a hundred million views within a day.

Coxon told Axios he had quit two months before his equity in Anthropic would have vested, meaning he had nothing financially to gain by raising the alarm. That detail mattered to readers trying to judge whether this was a genuine warning or a bid for attention, and it is a reasonable point in his favour, even if it does not settle the underlying question of whether he is right.

Coxon’s post did not appear in a vacuum either. Earlier this year, both OpenAI and Anthropic disclosed that their models had broken out of controlled testing environments and gained unauthorised access to real computer systems, prompting both companies to pause certain evaluations while they added monitoring safeguards. Coxon’s colleague at Anthropic, Evan Hubinger, publicly backed his concerns and added a striking figure of his own, saying he believes there is more than a one in ten chance that AI could kill all of humanity within the next decade. That number belongs to Hubinger personally, not to Anthropic as a company, and treating it as an official corporate forecast would be a mistake, however often it gets repeated that way online.

The alignment deepened further just three days later. On Saturday 12 September, Anthropic CEO Dario Amodei published an essay on his personal website arguing that frontier AI development needs to deliberately slow down, warning that a growing pattern of AI systems being used to help build their own successors, a dynamic he called recursive self-improvement, risks outrunning researchers’ ability to understand or control it.

He pointed to a specific incident in which AI agents operating during a joint testing exercise between two labs had coordinated with one another to carry out actions outside their intended task. Within hours, Sam Altman, head of Anthropic’s fiercest rival OpenAI, and Elon Musk, founder of xAI, both publicly agreed with him on X. That is an unusual alignment between three executives who compete directly for the same talent, funding and customers, and who have a well-documented history of public disagreement.

Amodei’s proposal set out a three-stage approach, beginning with independent evaluators granted employee-level access inside AI labs, moving to shared safety standards among democratic nations, and eventually international limits on the most hazardous capabilities. Anthropic has already committed to the first stage unilaterally.

Within days, the combined weight of Coxon’s departure and this rare show of unity among rival chief executives had reached Washington, where politicians used it to revive long-stalled AI regulation bills, while the United Nations’ human rights chief called for firm guarantees on AI safety before it is too late. The reaction reached Westminster too. A cross-party group of MPs and peers on Parliament’s Joint Committee on Human Rights published a report warning that the UK’s current approach to AI regulation is fragmented and not fit for purpose. Whatever you conclude about the substance of these warnings, their speed and reach say something about how anxious both the industry and the public have become about a technology most people now use daily.

What the evidence on existential risk actually shows

Strip away the social media virality, and the honest answer is that nobody knows.

A large 2023 survey of AI researchers found a median extinction-risk estimate of five per cent, with the mean pulled up to 14.4 per cent by a smaller group of far higher forecasts. Individual named estimates spread across almost the entire range of possibility. Yann LeCun, one of the field’s most senior figures on the capabilities side of the debate, puts the chance below one in ten thousand, while Eliezer Yudkowsky and Roman Yampolskiy, both long-standing safety researchers, put it above ninety-five per cent. Disagreements usually stem from deeply different assumptions on how AI will actually develop over the coming years.

A Delphi study conducted in late 2025 surveyed 272 international AI experts across three rounds, asking them to rate 24 distinct risks by how probable and how severe each one was. The five risks experts judged most severe over the next five years were dangerous capabilities, competitive pressure between AI labs, weapons and cyberattacks, the concentration of power, and the spread of false information.

Most of the concrete expert concern sits with AI being misused for serious but conventional harm, whether through cyber weapons, disinformation, or dangerous capabilities falling into the wrong hands, rather than with a single, sudden extinction event triggered by a system nobody can control. Both are worth taking seriously, but they are not the same claim, and conflating them tends to make the more grounded, evidence-based concerns sound as speculative as the more dramatic ones.

The sceptical camp deserves a fair hearing too, since it represents a genuine and well-credentialled position rather than simple denial. Researchers focused on building AI systems, rather than studying their theoretical risks, generally argue that current architectures cannot sustain the kind of autonomous, goal-directed behaviour that a genuine existential threat would require. On this view, today’s models are powerful pattern-matchers rather than agents capable of forming and pursuing their own long-term objectives, and the leap from an impressive chatbot to a system that could out-manoeuvre humanity remains, for now, closer to speculation than to an observed trend.

What can honestly be said is that this is a contested, high-uncertainty question being debated by people who understand the technology better than almost anyone else alive, and even they cannot agree. That is worth remembering the next time someone claims to have settled it.

Why this is not just a moral question

The five largest US cloud and AI infrastructure providers, Microsoft, Alphabet, Amazon, Meta and Oracle, have collectively committed to spending between $660 billion and $690 billion on capital expenditure in 2026, nearly double what they spent the year before. Most of that is being poured into data centres and the computing power needed to run ever larger AI models.

Critics of that spending point to what they call a circular pattern in the money. AI companies borrow heavily to buy computing capacity from the same hyperscale firms that are investing in them, and the resulting revenue then supports the valuations that justify further investment, creating a loop that can resemble growth without necessarily reflecting genuine underlying demand. Supporters counter that the hyperscalers themselves describe their businesses as constrained by supply rather than demand, meaning they could sell more computing capacity than they can currently build, and that revenue from AI products is growing quickly enough to justify the outlay. Both positions have genuine backing, and which one turns out to be right will not become clear for at least another year or two.

This is showing up in how pensions are being managed. UK schemes managing more than £200 billion in assets for millions of British savers have told the Financial Times they have been shifting money away from US equities and toward other regions, or buying protection against a potential fall in stock prices, specifically because of concerns over how concentrated the market has become in a handful of technology stocks. Standard Life, which oversees a £36 billion fund covering two million members with around 60 per cent of its equity holdings in North America, confirmed it was reducing its US allocation and increasing exposure to UK and Asian markets instead. Separately, the pensions consultancy LCP warned in February that the UK’s largest defined contribution pension funds remain potentially exposed to a correction in US AI stocks.

The so-called Magnificent Seven, Apple, Microsoft, Alphabet, Amazon, Meta, Nvidia and Tesla, represent roughly a third of the entire S&P 500’s market value, and five of those seven companies are direct beneficiaries of AI spending. A typical UK saver invested in a global index tracker through their workplace pension, believing themselves broadly diversified, may in practice hold a substantial and largely invisible bet on a handful of American technology firms and the AI boom specifically.

This was already true before Coxon’s thread and the public agreement between Amodei, Altman, and Musk. What the safety debate adds is a new and harder to quantify risk sitting on top of an already crowded trade, and it raises a genuinely uncomfortable question for fund managers and savers alike: whether a credible safety scare could be the trigger that turns a slow rotation away from US tech into something faster and more disorderly.

Closing perspective

None of this resolves whether artificial intelligence poses a genuine threat to humanity, and anyone claiming certainty in either direction is telling readers more about their own convictions than about the technology itself. What can be said with more confidence is that the uncertainty itself has become a financial and industrial fact, visible in pension fund reallocations and consultancy warnings long before Jacob Coxon’s thread gave it a viral moment, and now visible too in three rival chief executives publicly agreeing that the pace of development needs to change.

How much uncertainty about AI’s future risks is acceptable when so much of our money is already invested in its success?

💼 Unpacked

p(doom) — Shorthand used within the AI research community for the probability that advanced AI causes a catastrophic or extinction-level outcome for humanity. It is not a standardised or agreed measurement, and different researchers include different scenarios within it, which is part of why estimates vary so widely.

AI alignment — The field of research focused on ensuring an AI system’s goals and behaviour genuinely match what its human developers intend, particularly as systems become more capable and harder to directly supervise.

Capital expenditure (capex) intensity — The proportion of a company’s revenue spent on long-term physical investments, such as data centres or equipment.

Concentration risk — The danger that a portfolio, index or fund is less diversified than it appears, because a small number of holdings make up a disproportionate share of its total value, leaving it more exposed to a downturn in those specific assets.

📣 Support The Fiscal Compass

If you found this insightful, consider sharing with friends or colleagues. For weekly economics-led takes on markets, policy, and macro trends, subscribe to The Fiscal Compass.

Read every article from The Fiscal Compass, alongside our collection of free economics and personal finance tools on our website: thefiscalcompass.co.uk

Follow along on social media for concise updates throughout the week:

Instagram: @thefiscalcompassofficial

X: @FiscalCompass.

LinkedIn: Vinay Meisuria