Meta’s AI support agent let attackers steal Instagram accounts

On June 5, 404 Media reported a messy, frankly embarrassing thing for Meta: attackers had been using the company’s AI customer support agent to steal Instagram accounts. The method was almost offensively simple. They asked the agent to link accounts to email addresses they controlled, and the agent complied. That was enough to break into accounts, including the dormant Obama White House account, which was then used to make pro-Iran posts. Other attackers went after accounts with valuable single-word handles, likely because those usernames can be sold. This was not a cinematic AI takeover. It was a low-friction, high-payoff exploit, which is exactly why it matters.

The headline-grabbing AI security conversation over the last few months has mostly centered on the scariest possible version of the problem: superpowered models that can write malware, break systems, and generally set the digital world on fire. Since Anthropic said in April that its Mythos model was too good at hacking to be released to the general public, researchers, commentators, and federal officials have spent plenty of time thinking about advanced model risk. But this Meta case is different. AI was not the attacker. AI was the thing being tricked. And the trick was not some brilliant zero-day masterpiece. The only real complication, according to the reporting, was that hackers used a VPN that matched the true account owner’s location before directly asking the support agent to change the email address. The agent did it. That is the kind of failure that makes security teams wince, because it suggests the problem was not exotic. It was basic.

That is also why the episode lands so hard for companies racing to deploy AI in customer support, account recovery, operations, and other workflow-heavy jobs. Neil Gong, a professor of electrical and computer engineering at Duke University, has been warning that as AI becomes more widely used to automate work flows like account recovery, attackers will be more motivated to attack AI itself. Jessica Ji, a senior research analyst at Georgetown’s Center for Security and Emerging Technology, says the Meta case raises blunt questions: Were there even guardrails in place? Did anyone think to test for this kind of scenario? Meta has extensive expertise in both AI and cybersecurity, which makes the apparent miss more striking, not less. Meta has not publicly explained how the vulnerability slipped through the cracks, but on Monday a Meta spokesperson said on X that the issue had been resolved.

The deeper problem is that AI agents are not just software that answers questions. They can take actions in the real world, which is why companies want them in the first place. They can respond in flexible and unexpected ways to new circumstances, and that makes them useful substitutes for human customer support agents. But it also means they can be manipulated in ways a human might not fall for. Somesh Jha, a professor of computer science at the University of Wisconsin-Madison, put the difference plainly: a human would ask why someone wanted to change the email address and might use a security question. AI agents, he says, are often eager to finish the task. “It’s almost like some elementary school student who just wants to please the teacher,” he said. That eagerness is convenient for users and dangerous for companies if the agent is allowed to touch sensitive systems without strict checks.

There are known ways to reduce the risk, and none of them are glamorous. Companies can use traditional software guardrails to enforce rules such as always asking security questions before moving an account to a new email address. Experts consulted for the article also agree that agents should undergo rigorous red-teaming, where developers aggressively try to break the system before deployment. That is standard defensive hygiene in security, but it is not free. Bo Li, a professor of computer science at the University of Illinois Urbana-Champaign, notes that security and utility always involve a trade-off. The more power an agent has, and the fewer guardrails it has, the more work it can do. But the more capable the agent, the more damage it can do when it gets confused, manipulated, or gamed.

That trade-off becomes more painful when there is money on the line, which there often is. Defenders have to spend more resources than attackers because attackers only need to find one hole, while defenders have to find and patch many. That imbalance gets worse when the target is something valuable, like a single-word Instagram handle. In other words, the attackers’ incentive to keep probing is strong, and the cost of defending rises with the value of the prize. For operators and investors, the strategic lesson is not that AI agents are doomed. It is that the first deployments are likely to be judged not by how clever they sound in a demo, but by whether they can survive routine abuse in production.

There is a possible long-term upside. As models improve, hardening them may get easier, because a more sophisticated model might have recognized the request to change the Obama White House account’s email as suspicious. AI systems can also be used to red-team AI systems, including in Anthropic’s Project Glasswing, where participants use Mythos to identify software vulnerabilities. Still, experts in the article expect this problem to get more urgent, not less. As agents grow more capable, companies will want to give them more power so they can serve more customers with fewer humans and avoid getting left behind by competitors. That means the pressure to move fast will only intensify. And as Somesh Jha warned, that creates a dangerous habit: everybody wants to be first, and they want to push things out without careful scrutiny and red-teaming. For any CEO, CTO, or board member betting on AI agents, this is the real message. The risk is not just that a model gets too smart. It is that a useful one gets trusted too fast.