AI built 40,000 bioweapon molecules in 6 hours. Now CEOs are scared.

The number is as stark as it gets: in less than six hours, an AI model designed for drug discovery generated 40,000 molecules with potential as chemical warfare agents, some engineered to be even more toxic than known nerve agents. That 2022 result from Collaborations Pharmaceuticals has become the touchstone for a fear now being voiced by the highest-profile executives in AI. Over the past two weeks, Anthropic CEO Dario Amodei argued publicly that AI carries serious risk and that progress should be slowed, and OpenAI CEO Sam Altman replied on X, "I agree with Dario that we need to pace the frontier." The exchange followed researcher Jacob Coxon's departure from Anthropic, where he charged that both it and OpenAI were not acting responsibly, posting, "The people building AI earnestly believe that it could kill us all by the end of the decade." Anthropic's Evan Hubinger agreed, putting the probability of AI ending humanity at more than 10 percent within a decade. One of the pathways they fear is AI-aided bioweapons.

That fear is not abstract. A bioweapon could be a gene-targeting lethal virus, a fungus that destroys crops, or a tasteless toxin slipped into a water supply. The concern is that AI tools can help generate such agents. The 2022 experiment proved it: the molecule generator, built to find drug candidates, was trivially repurposed. The authors themselves wrote, "Without being overly alarmist, this should serve as a wake-up call for our colleagues in the 'AI in drug discovery' community." That wake-up call reached David Magnus, a Stanford professor of medicine and biomedical ethics who has studied biosecurity risks since the late 1990s. "That was very scary to me," he said. "Of course, everything since then has just sort of blown up."

Today, large language models are trained on the knowledge of "almost every scientist who ever lived on this planet," according to Dunja Sabra, a biosecurity researcher at the University of Hamburg. Those models can provide instructions and video training for experiments. Combined with more accessible gene editing and synthetic biology tools, plus the DIY biology movement that has enabled home labs, the barrier to a determined actor has collapsed. "The chances are that someone determined would succeed eventually," Sabra says. This is not a distant threat; it is a capability that already exists.

Safeguards are in place, but they are not ironclad. DNA suppliers typically screen orders for suspicious sequences. Responsible researchers run "red-teaming" (independent scientists probing for misuse) and "blue-teaming" (developing mitigations). AI companies have tweaked their models to withhold dangerous scientific information. But Anthropic's own report last week acknowledged that users had attempted to use its models to explore making the chikungunya virus more transmissible, creating a more dangerous bird flu, and building an "atlas of venom toxin peptides." Magnus describes a constant back-and-forth: "We have to build better surveillance and screening tools, [but] AI is really good at figuring out ways around them." His conclusion: we will likely need AI to restrict AI.

Not everyone agrees on the risk level. At a recent media briefing, biologists at Imperial College London argued that AI tools are not good enough to fully develop bioweapons, and that testing new pathogens still requires difficult, time-consuming human work. Some believe current guardrails are sufficient. Wendy Barclay, a professor of infectious disease at Imperial, pointed out that the greatest pandemic risk today is not a bioweapon but already-circulating pathogens - H5N1 bird flu, which has killed millions of birds, spread through US dairy cattle, and was last month detected in captive mink at a Utah farm.

Sabra, however, thinks five to 10 years ahead. Countries should strengthen health-care systems, prepare antidotes to known toxins, and stockpile medicines. "We need to be prepared," she says. MIT biologist Kevin Esvelt, who invented gene-drive technology and its safeguards, added a stark warning on X: a large language model had "disclosed a novel form of bioweapon that I hadn't realized was possible." He pleaded, "Please, for the love of God, children, the future of humanity, or whatever you consider holy, let's err on the side of caution here."

For executives and boards across biotech and AI, the strategic stakes are immediate. The same tools that promise breakthroughs in drug discovery and synthetic biology carry a dual-use risk that regulators, investors, and insurers are beginning to price in. The 2022 experiment is not a hypothetical; it is a documented proof point. Companies that fail to embed red-teaming, DNA screening, and AI guardrails into their product roadmaps are exposing themselves to reputational, legal, and existential risk. Those that proactively invest in biosecurity - both defensive detection and offensive mitigation - will set the standard for a regulatory landscape that is surely coming. The question is not whether AI can enable bioweapons; it already has. The question is whether the industry will act before a determined actor does.