German researchers show medical AI can reveal whether a patient was in training

A Nature paper published Wednesday makes a very uncomfortable point: medical diagnosis AIs can be tricked into revealing who their training data included. In tests across seven medical AI datasets, German researchers found that “individual patients targeted by such attacks can be identified with near-perfect attack success.” That is the headline consequence, plain and simple. And it directly clashes with how many teams evaluate these systems for privacy safety, which often measures attack success in aggregate across records rather than at the individual level.

So what is the practical problem for decision-makers? If a patient’s data was used to educate a discriminative AI model, an attacker may be able to infer whether that person’s datapoint was part of the training set. The researchers warn that this can spill sensitive medical history and diagnoses, even if the model itself is deployed as a black box. In other words, the system is not just “predicting disease.” It can also become a tool for confirming membership in a dataset, which is a different kind of risk with different worst-case impact.

The paper focuses on discriminative AI models, the kind that classify data and make predictions about new inputs based on their training sets. Those models are the target of membership inference attacks (MIAs). Here is how MIAs work at a high level: the attacker queries the model with patient data they have reason to believe is related to a particular person, then observes the model’s confidence. The attack leverages a pattern that the researchers describe: the AI tends to be more certain when the input resembles data it has seen during training. In the researchers’ framing, this means the attacker does not need the model to say anything meaningful about diagnoses to cause harm; the model’s confidence score can function as a membership signal.

Just as important, the researchers’ results push back on an assumption that often shows up in privacy discussions: that anonymization or lack of exact identity mapping would neutralize the danger. The paper says the attackers did not need access to a full patient record to run effective MIAs. Knolle, the Technical University of Munich AI in Healthcare and Medicine chair and the paper lead author, told The Register that an attacker with partial access can still successfully conduct MIAs. He also emphasized that dataset inputs in their study were anonymized, yet MIAs were largely error-free at the individual patient level. That combination matters because it suggests the risk is not only about sloppy data handling in the dataset itself. It is also about how the model learned from that dataset and how confidence behaves when it has seen something before.

The datasets examined in the analysis span images, ECG records, and general electronic health records. The researchers argue the privacy evaluation approach is miscalibrated. Their statement, as quoted in the report, is that the fact MIAs can achieve near-perfect success rates for individual patients is not adequately captured by a standard evaluation protocol that measures attack success across records in aggregate. Aggregate metrics can make the system look safer than it is for the one person an attacker cares about. That is an audit problem, not just a research problem: if your test averages the risk, the worst-case individual exposure can be effectively hidden.

The paper also adds a grim “who gets targeted” layer. Patients are generally easy to identify in the underlying contexts the researchers studied, and underrepresented groups can be even easier to finger. Underrepresentation can show up in multiple sensitive categories, including race, insurance status, sex, the protocol used to conduct medical imaging, and certain disease statuses. The reason is intuitive even if the result is devastating: outliers in training distributions can make membership signals sharper. Knolle specifically told The Register that privacy risks from MIAs become more severe as a model’s training cohort becomes more specific. He added an example scenario: membership in a training dataset could reveal that someone has a dormant genetic condition such as Huntington's disease, depression, or attended a specific, specialised treatment clinic. The strategic takeaway for boards and founders is that the harm is not only the fact that data is “leaked,” it is what membership implies.

There is also a scale effect. The report notes that the magnitude of this change in patient-level risk was previously unknown in larger models, and it states that, as datasets get larger, the easier it is to expose records. This matters because many medical AI teams are already trending toward bigger training sets and more specific models. If privacy risk changes with scale and specificity in ways standard audits do not capture, then the security posture cannot be treated as a one-time checkbox when launching a product or publishing a paper.

The researchers are not only raising alarm. They outline what they want to change. Knolle said his hope is that the medical AI community will take privacy risks seriously and use risk mitigation techniques when necessary. The paper’s recommendations include differential privacy frameworks designed to mathematically guarantee training data remains anonymous, and a push for privacy audit standards that consider individual-level data rather than just aggregate privacy risks. The report also notes an alternative mitigation direction: compiling datasets so underrepresented groups are better represented. And in a nuance that should steer how leaders think about severity, Knolle pointed out that in some situations, a successful MIA represents a small or negligible privacy violation, particularly when models are trained on large general populations where healthy and diseased individuals are represented in sufficient numbers.

For executives, the stakes are bigger than technical compliance. This research reframes “privacy” as a property of the model plus its evaluation protocol, not just a property of the dataset’s labeling. As medical AI moves from research prototypes into clinical workflows and procurement pipelines, boards will need to ask whether their vendors and internal teams can demonstrate that individual-level privacy risk is understood and tested. In a world where healthcare data breaches are common enough to be expected, the question is no longer whether an attacker can get data. It is whether your model turns that data into a membership oracle.