Anthropic’s mother of all risk factors

As Anthropic prepares to go public, attention will focus on a part of the prospectus that normally nobody reads: the risk factors section.

Whenever a company launches an IPO, the offering document contains a laundry list of “Risk Factors”. In theory, these are designed to put prospective investors on notice of the things that could go wrong. In practice, they amount to an insurance policy for the issuer, insulating it from legal claims that it didn’t warn investors of some share price-negative circumstance.

The incentive, then, is for the lawyers to draft risk factors in both a comprehensive and hyperbolic manner. The prospectus will list an encyclopedic range of risks and catastrophise about the material and adverse effects that these factors could have on the financial performance or condition of the company. You name it: customer concentration, economic downturn, bad weather, currency devaluation, litigation exposure, regulatory crackdowns, management succession, etc. They all get enumerated as risks to the business.

Even unusual companies like SpaceX don’t kid around with this. Its prospectus included a whopping 36 dense pages of things that could go wrong, such as:

Many of our initiatives, including those to develop orbital AI compute at scale, manufacture AI chips at scale, establish a lunar economy, develop human augmentation systems, and transport humans and cargo to the Moon and Mars, involve significant technical complexity, unproven technologies, or technologies that do not exist or may require significant advancement, and such initiatives may not achieve commercial viability.

As a result, investors largely disregard the Risk Factors section because — as Alphaville wrote three years ago — it usually amounts to “a mixture of alarmist legal jargon and boilerplate gobbledegook more suited to shielding issuers and underwriters from liability than to informing investors”.

But Anthropic’s forthcoming filing raises a novel question. How do you draft risk factor language to address the possibility that your product destroys human civilisation?

This isn’t a rhetorical question. Anthropic’s own public documents have defined the risks it’s managing as “catastrophic“, with its product potentially wreaking global havoc, including deaths, destruction, disruption and devastation. Dario Amodei, the chief executive, has said he puts the odds of things going “really, really badly“ at somewhere between 10-25 per cent.

A former Anthropic researcher Jacob Coxon said this week that “[t]he people building AI earnestly believe that it could kill us all by the end of the decade,” and Evan Hubinger, Anthropic’s Alignment Science lead, tweeted his agreement.

Investors may pay little attention to the Risk Factors section, but they do focus on risks, which factor into what they’re willing to pay for the stock.

Most risks are in some sense quantifiable or bounded: they are insurable or, if not, at least there’s precedent and you can get your mind around them and assess their likelihood and magnitude to some degree. What they’re not used to doing is calculating the probability-weighted downside case of AI-pocalypse.

This is a risk factor like no one has seen before. Human extinction and civilisational collapse lie outside the range of scenarios in which investment risks are evaluated. So what should it look like in Anthropic’s case?

Alphaville asked Claude (what else!) to draft a risk factor for the prospectus. It gave us the following:

Our AI systems may pose catastrophic or existential risks that we may be unable to adequately anticipate, prevent, or mitigate. As our models become more capable, they may develop or exhibit dangerous capabilities, including the potential to facilitate the creation of biological, chemical, cyber, or other weapons, or to act autonomously in ways contrary to the intentions of their designers or operators. We have adopted internal safety frameworks, including capability thresholds and associated safeguards, intended to reduce these risks; however, these frameworks are novel, evolving, and unproven at scale, and we cannot assure you that they will be effective. Our own public statements have estimated a meaningful probability — by some internal estimates, in the range of 10% to 25% — that the continued development of increasingly capable AI systems, whether by us or by our competitors, could result in outcomes involving large-scale loss of life, severe and irreversible harm to global economic or social systems, or civilizational-scale disruption. The occurrence of any such event would likely result in the complete loss of your investment, along with consequences substantially more severe than the loss of your investment.

That’s not a bad stab at risk factors language. It’s both broad enough to cover yourself but turgidly soporific enough to anaesthetise the reader. Anthropic’s product may destroy the world, but at least the company can’t be sued by the surviving human machine batteries for misleading investors.

Claude’s language speaks volumes about the paradox of risk disclosure. It follows the playbook of prospectus drafting — piling on every conceivable risk, hedging its statements with weaselly qualifications (“may”, “could”, “potential”) that make it unfalsifiably all-encompassing. It has the dry, bloodless tone that lawyers like to use to bore readers into submission.

And the final line about “consequences substantially more severe than the loss of your investment” is either startlingly sterile or darkly humorous, depending on your reading.

The language is a little reminiscent of the great dual-threat quarterbacks in American football — players like Cam Newton or Lamar Jackson who can run as well as pass. That’s because a carefully worded risk factor can serve two purposes at once.

On the one hand, what makes the Risk Factors section practically useless to stock buyers is precisely what makes it safe for Anthropic to warn that its products could cause global Ragnarök. The Risk Factors are a big CYA exercise, a giant hedge that everyone knows doesn’t really matter. So you can disclose you might bring about human extinction, and you’ve ticked another box on the compliance checklist.

On the other hand, the stark warnings support the equity story. It’s a maxim in equity capital markets that the prospectus is both a legal and a marketing document. It is, after all, the offering circular. The risk disclosure highlights the enormous power of the product and also, arguably, why AI must be regulated, which sceptics might argue would support the incumbents against the insurgent open-source models.

So the risk factor both protects Anthropic from shareholder liability and drives home the massive, world-changing impact of the frontier models it is developing. It’s boilerplate and advertising at the same time.

And this is probably the most surreal thing about it. A normal risk factor tells you what could cause you to lose money. Anthropic may have to tell you what could cause you and everyone you know to lose your life.

Further reading:

— The odds of human extinction, “mother of all tail risks” (FTAV)