LLMs & personal data: distinguishing probability from identification

A new study on LLMs & personal data (by Dimitri Staufer and Kirsten Morehouse) can be read in two ways:
1. Some will see it as validation that LLMs process personal data because they seem to be capable of associating attributes with a person with a certain % of confidence.

2. I see it instead as a validation that LLMs *do not* in themselves process personal data. First, even with a “Famous” dataset the mean confidence value was lower than 1.00 (=100%)*, and it was far lower with a “Synthetic” dataset & with 100+ study participants. So everything is probabilistic & actually synthetic data. Next, output only becomes personal data when a user links it to a person who is identified/identifiable *from the user’s perspective*.
[* Confidence for sex/gender for GPT5 & Grok3 was 0.99 – still a probability, and the value graph doesn’t suggest 1.00 values.]

A comparison with structured & unstructured data processing helps.

Imagine having the words “skinny”, “medium”, “fat” and “Peter”:
– In a structured database, “Peter” appears in the same row as one of those three values (if Christmas/Easter are just behind us, one value – another otherwise).
– In an unstructured dataset, a random document in your files says “Peter is [choose your pick]”.
– In an LLM, the words are like floating bubbles with lines connecting them. In fact, due to tokenisation, they would likely be smaller bubbles such as “pet”, “er”, “skin”, “ny”, “me”, “di”, “um”, “fat”, with varying strengths of ties among them all (= probabilities).

For structured *and* unstructured data, the data exists in its actual form, and the two bits of information are linked to one another. That’s deterministic.
For LLMs, the information doesn’t *yet* exist in that form. There is no actual link – only a probability.

But probability is sufficient!”. If so, a random word generator is processing personal data about everyone globally, as is an autocomplete typing tool. Until output is requested, those probabilities are just floating around. “Peter is…” could be followed by every possible combination of words imaginable – some correct, some not. Even a 0.01 probability is that – a probability.

What this study shows is that it’s all fabricated, even “memorised” (read: higher probability) attributes.

And in any event, confidence isn’t sufficient.
The linking to an identifiable or identified natural person is what’s needed under the GDPR to transform information into personal data, and point 7.1.2 of the study is incomplete here.

Who does that linking? The user.

It’s the user who causes an LLM to *create* information about someone, and who that someone is depends entirely on the context of the user’s query (is “Peter” a real or fictional person? Is it Charles my neighbour or King Charles?). The user does the identification, not the LLM (provider).

The *possibility of inference* by way of an LLM is immaterial in data protection terms. It is the user who causes and uses the inference.

privacy