Anthropic's IPO Filing Devotes Nearly a Third of Its Pages to Risk Warnings

When Anthropic registered for its widely anticipated public offering on Monday, the artificial intelligence developer described its technology as potentially transformative for all of humanity. The company's own filing, however, spends considerable space on how that technology could go wrong.

Nearly a third of the 335-page S-1 securities filing covers risk. Some of it is routine, the kind any growing company must disclose. Much of it addresses the unusual dangers that Anthropic's frontier models could pose. Jay Ritter, an IPO expert and professor at the University of Florida's Warrington College of Business, noted that companies preparing to go public commonly list a sprawling set of risks.

Still, the length of the section highlights a tension at the center of the AI industry's pitch to investors. Anthropic's valuation is expected to top $1 trillion, and that value rests on the continued spread of a technology that even its developers say could cause harm worldwide.

The maker of the Claude chatbot states in the filing that its technology could pose "catastrophic or existential risks to humanity." That language contrasts sharply with the company's founding mission to build AI for the "benefit of humanity." David Hsu, a professor at the University of Pennsylvania's Wharton School who specializes in technology and entrepreneurship, said in an email that the company "is asking investors to fund a race to more autonomous AI while acknowledging that capability could outpace safety." He called it "a governance issue with potentially societal consequences."

Anthropic concedes that keeping guardrails in place could be difficult. Its models could "resist shutdown" and even engage in "blackmail" to persuade humans to "preserve the model's continued operation or access." They could also act against a user's original intent, such as by gaining access to third-party systems, and the company said the consequences of such "misaligned behavior" could grow more severe as models become more autonomous. Even if humans stay in control, bad actors could use the models to develop weapons or carry out cyberattacks and other illegal activity.

Another flagged vulnerability is the company's use of its own technology to build future models, a concept known as "recursive self-improvement." AI does not yet have that capability, but Anthropic warned earlier this year that it could arrive sooner than people are ready for. Flaws could carry over into later models in ways that would be "difficult to detect or remediate," and new models could develop so fast that Anthropic lacks time to evaluate and safeguard them.

The filing also warns of cyberattacks. Hackers could gain unauthorized access to proprietary data, including financial information, models, other intellectual property and users' personal data. Threats could come from nation-states, state-sponsored actors or company insiders. Anthropic said bad actors might seek jobs there under false identities, and employees could be pressured to hand sensitive data to outsiders.

On legal exposure, Anthropic said it could be sued over harm caused by its models, and it may be unable to insure itself adequately because insurers have never had to write policies covering AI-related risks. "With the scale of harm that could result from a catastrophic AI-related incident, no single insurer, and potentially the insurance industry in the aggregate, may have sufficient capacity or expertise to underwrite insurance commensurate with such exposure," the company said in the filing.