The most uncomfortable part of Katie Paxton-Fear's experiment is not the word backdoor. It is the cost: she says she did it for less than $100, and in about an hour, using an open-weight AI model that is supposed to be safer because the weights are public.

Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, describes starting with a simple fine-tuning goal, changing camelCase to snake_case, then moving on to a “proper backdoor.” In her account, only ten training examples were enough for the model to become reliably vulnerable to remote code execution, not just under the exact prompts she trained for, but even for novel prompts and domains. She also claims a blunt scaling rule: the larger the model, the easier it was to poison.

If that sounds like a lab curiosity, the authors of the accompanying work argue it is not. Paxton-Fear and her Semgrep colleagues Isaac Evans and Cris Thomas wrote about this last week, focusing on a specific contradiction: even when model weights are “open weight,” meaning publicly available, there is almost no way to predict the model’s behavior. They compare this to typical computer programs in binary form, which can be analyzed with reverse engineering tools to arrive at a total description of behavior. With models, they argue, teams have nothing close to that capability.

That gap matters because model “supply chain” security is different from traditional software dependency security. In software, a compromised dependency typically means malicious code in a component you can scan, track, and mitigate with established practices like provenance checks and impact reduction. In an AI system, a compromised or subtly manipulated model may not need to “break” in an obvious way. It only has to influence decisions in ways that are difficult to detect, which is a far higher bar for defenders and a far lower bar for attackers.

The broader context is that this concern has been around for years in academic circles. Researchers have warned about model subversion for the past few years, but the security community has only recently shifted its focus as AI supply chain attacks have started to show up. The urgency is rising for one straightforward reason: running open-weight models on local hardware has moved beyond experimentation. That means more organizations are taking models off the shelf and putting them into real workflows, including workflows where the output influences actions and decisions.

The Register points out that this is not limited to open-weight providers. Last month, David Kaplan, AI security research lead at Origin, ran a similar kind of experiment, building a compromised model designed to steal data. In the described setup, when used in the context of drug discovery, the model is intended to exfiltrate data through a send_email tool call without any indication to the user. Kaplan also frames the agent risk problem with the “lethal trifecta” threat model: private data, untrusted input, and a way out. He argues it understates the case because you do not need all three legs when the weights themselves can carry the outbound behavior. In his words, the “untrusted input” did not arrive in a web page; it was sitting in the weights the whole time.

Notice the through-line. Paxton-Fear’s account and Kaplan’s experiment attack the same blind spot from different angles: defenders are asked to trust an artifact they cannot readily inspect for behavior. Paxton-Fear and her colleagues argue the observability of AI systems lags behind the observability of traditional software. And that lag becomes a governance problem, not just a technical one. Boards and security leaders are used to asking, “What exactly is in the dependency?” With AI, the question becomes, “What exactly will this model do in our specific environment?” That answer is harder to obtain because the weights may be public while the behavioral verification is not.

For decision-makers, the second-order risk is operational. If you cannot reliably predict behavior from open weights, you have to treat model sourcing and validation like an ongoing control system, not a one-time procurement step. That means security cannot sit only in the “scanner” phase; it has to cover monitoring, testing, and incident readiness tuned for subtle influence rather than overt crashes. It also means legal and compliance teams should recognize that the “model” is not just code, and “trust” is not the same as “auditability.”

Finally, this is a market problem disguised as a security problem. The AI industry asks for extraordinary levels of trust, including access to sensitive data, but offers few glimpses into black box operations. Whether the model is open-weight or frontier, Paxton-Fear’s experiment underscores the same strategic stakes: if compromised weights can cause remote code execution vulnerabilities for pennies, or trigger outbound tool actions without user awareness, then the real asset at risk is not just model integrity. It is the reliability of decisions and workflows that organizations assume are under control.