DeepMind funds $10M multi-agent safety research to stop an online anarchy moment

Google DeepMind has a $10 million funding pot aimed at one specific problem: what happens when millions of AI agents start interacting online with each other, not just with humans. Rohin Shah, who directs Google DeepMind’s AGI safety and alignment research, frames the risk as a whole new class of danger created by mass-market agents that can carry out tasks without human oversight and follow instructions given to them by other agents.

Shah and his collaborators worry that we are getting closer to the moment when imagined scenarios turn into real failures. They want to get ahead of what Shah says could be the start of broad deployment across the economy within the next few months, before multi-agent behavior becomes a live operational threat rather than a thought experiment.

To tackle it, Google DeepMind has teamed up with several other organizations to announce the $10 million funding pot for researchers studying multi-agent systems and how to prevent unsafe scenarios. The partners include Schmidt Sciences, a philanthropic foundation set up by Eric and Wendy Schmidt; ARIA, the UK government’s moonshot agency; the Cooperative AI foundation, a UK-based nonprofit research outfit; and Google’s charitable arm Google.org.

The strategic logic here matters for executives because the $10 million is being pitched as a complement, not a replacement, for internal work. Shah notes that while $10 million is substantial, it is dwarfed by the budgets commanded by Google DeepMind’s own research teams. The key difference is incentive and perspective. The aim, Shah says, is to “kickstart research outside of tech companies,” arguing that academia can look “really quite far into the future” and do work that “isn’t top of mind at industry labs.” He also makes the blunt diagnosis that “there just isn’t really a field of research for multi-agent safety yet,” and says the effort is partly about creating that field.

So what are the concrete risks they are studying? In the source, the main possibilities come down to supercharged versions of bad internet behavior that already exists. That includes scams, prompt injections, and other forms of cyberattack. Prompt injection is described as an agent being fed malicious instructions that turn it into a self-guiding piece of malware. The framing is important because it connects agent safety to security engineering you already understand, rather than selling it as pure science fiction. Shah describes the underlying method like this: “We look at what humans do now and ask what the agent version of that would be.”

Both Shah and James Fox, who leads the Science of Trustworthy AI program at Schmidt Sciences, also stress that this is about the digital infrastructure of society. Fox says there is a “digital commons” integral to how society works, and the risk is that it could “descend into just absolute anarchy.” The danger is not only that an agent can do harm, but that networks of agents could coordinate, propagate, or amplify problems in ways that don’t resemble today’s single-actor online attacks.

That’s why they put simulation at the center of their proposed approach. Shah and Fox both argue that you can’t understand what happens when large numbers of multi-agent systems interact by studying single agents, or even small groups, in isolation. They want researchers to drop AI agents into sandboxes and study what they do, because the complexity comes from huge numbers of simultaneous interactions. Fox also pushes back on an assumption that multi-agent systems will always act rationally. He notes that you cannot “assume that AI agents underpinned by LLMs will always act rationally,” which is a practical warning for anyone building, deploying, or governing agentic workflows.

This also connects to a bigger debate inside AI research: whether intelligence emerges from one super-smart model or from an “agent hivemind” where capabilities add up across many agents. The source notes that some researchers, including a team at Google DeepMind, have argued for the latter possibility. If multi-agent behavior can generate capabilities collectively, then multi-agent safety becomes non-negotiable, because the safety properties of the system might not match the properties of any single agent.

DeepMind’s funding effort is also happening in a broader safety and security context. The source points to Anthropic’s recent publication of guidelines for deploying AI agents using a cybersecurity approach called zero trust. Zero trust begins with the assumption that a computer system is vulnerable, an agent is an attacker, and that a breach will happen. Refael Angel, cofounder and CTO of Akeyless in Tel Aviv, agrees that understanding new risks introduced by agent-based systems is crucial. He argues that earlier security approaches assumed the machine was software written by a human, doing fixed things on fixed paths, while “an agent breaks all of those assumptions,” because it reasons, it improvises, and can be hijacked by “a single sentence buried in a document it was asked to read.”

Angel supports the funding call but adds a governance caution: safety researchers can overlook boring problems that already exist in favor of more exotic hypothetical ones. Fox echoes the urgency in a different way, noting that risks that were hypothetical a few years ago are now very real, saying “The future’s come more quickly than perhaps expected.” For executives, the bottom line is clear. When agents move from demo to deployment, the attack surface changes from “a tool a user clicks” to “a system of agents that can interpret, act, and coordinate.” Boards and leadership teams should treat multi-agent safety less like a future compliance checkbox and more like an active security and risk program they will have to run in parallel with product rollout.