Entity types differ in harm, and some are personal only together

The first labelled corpus a team builds usually goes wrong at the guideline stage, and the symptom is inter-annotator disagreement — two people labelling the same documents marking different spans — that nobody can adjudicate. Two people are given 400 support transcripts and an instruction that reads, in full, "highlight any personal data." One of them highlights job titles, employer names, and the city a customer mentions living near. The other highlights only names, emails, phone numbers, and account numbers. They agree on 68% of spans, which sounds like a training problem, and it is not: the instruction did not tell them what personal data means, and the two defensible readings of that phrase differ by roughly a third of the annotations.

Fixing the guideline requires deciding, in advance, which categories of string you care about and how much you care about each. That decision cannot be deferred until after the measurement, because the measurement's units come from it. A recall number — the share of the personal data actually present in a corpus that the detector managed to find — is always recall on a class, and if the class is "personal data" as an undifferentiated whole, the number you get back is a weighted average over whatever mix your corpus happened to contain, which tells you nothing you can act on.

What makes one entity worse than another#

Harm is not a single quantity, and the ranking is more defensible when you keep the contributing factors apart rather than assigning a vibe score.

Directness is the first. An identifier picks out a person by itself — a national identity number, a passport number, an email address, a full name in a small population. An attribute describes a person without picking them out: a job title, a city, a purchase amount. The distinction is the one that k-anonymity research — the literature on when a released record can be narrowed down to fewer than k possible people — calls direct identifiers versus quasi-identifiers, and it matters because the two fail differently, which the last section of this chapter is about.

Irrevocability is the second, and it is the one engineers underweight. A payment card number that leaks is bad and it is also fixable: the customer calls the bank, the card is reissued in three working days, and the leaked string is dead. A national identity number cannot be reissued in most jurisdictions without a court process, so a disclosure today is a disclosure that remains live for the person's whole life. A date of birth is worse still on this axis, because it cannot be changed at all. Ranking by "how sensitive does this feel" tends to put card numbers at the top because fraud is vivid; ranking by irrevocability puts them below identifiers the person is stuck with.

Consequence on disclosure is the third, and it is the one that depends on who the person is rather than what the string is. A home address is a routine piece of data for most customers and a safety-critical one for a customer who has a protective order, and your pipeline cannot tell the difference. Article 9 of the GDPR — the European Union's General Data Protection Regulation, the law that governs personal data about anyone in the EU regardless of where your servers sit — singles out a set of special categories, and they sit at the top of this axis. Health, sexual orientation, religious belief, trade union membership, biometric and genetic data are all there because the consequence of disclosure is discrimination or physical risk rather than inconvenience, and no remediation reverses it.

Regulatory category is the fourth, and it is a constraint rather than a harm. A primary account number, the long digit string on the front of a payment card and usually abbreviated to PAN, brings PCI DSS scope with it — the Payment Card Industry Data Security Standard, a contractual regime the card networks impose on anyone who stores or transmits card data — which changes what systems may hold it and what evidence you owe an auditor, independent of how bad the disclosure would be. Treating a regulatory boundary as a harm level conflates two things that need separate decisions, because a low-harm string in a regulated class can still be the one that stops a project.

Turning two recall numbers into a decision#

A ranking on its own still does not produce a decision, because a category you care about a great deal and that appears twice a year is not the same exposure as one you care about moderately and that appears in every third transcript. The quantity that combines them is the expected number of missed occurrences over a period, which is the traffic volume times the prevalence of that entity in the traffic times the detector's miss rate.

def expected_leaks_per_month(
    docs_per_day: int,
    occurrences_per_1k_docs: float,
    recall: float,
) -> float:
    occurrences = docs_per_day * 30 * (occurrences_per_1k_docs / 1000)
    return occurrences * (1 - recall)

Run it over a summarisation pipeline that sends 6,000 escalated transcripts a day to an external provider, with prevalence taken from the same reviewed corpus the recall measurement uses, and the picture stops being abstract.

Two things fall out of that table which the raw recall numbers hide. The 71% on national identity numbers is a stop even though only 214 occurrences a month get through, because the tier-1 budget is zero and irrevocable identifiers do not get an allowance. The 96% on work email addresses is a ship despite a comparable absolute count, because the tier-3 budget is nonzero and a work email address in a B2B support transcript is a string the customer typed into the ticket themselves. Recall alone would have ranked email as the safest entity and national ID as merely the worst of a bad set; the tiers and the budgets are what turn the same two numbers into two different actions.

The budgets themselves are the part that looks arbitrary and is not, if you set them the way you set an error budget. A tier-3 budget of 500 missed occurrences a month is a statement that you are prepared to be wrong at roughly that rate, that you will notice when you exceed it, and that exceeding it triggers work rather than a discussion. Setting it to zero for every tier feels safer and produces a system nobody ships, which is a decision too, just an unstated one. Chapter 4 prices what pushing every tier to zero actually costs.

Encoding the ranking in the pipeline rather than in a spreadsheet is what keeps it honest, because then a new entity type has to be assigned a tier before it can be handled at all.

entities:
  national_id:      { tier: 1, action: block,    budget_per_month: 0 }
  payment_card:     { tier: 1, action: block,    budget_per_month: 0 }
  health_condition: { tier: 1, action: block,    budget_per_month: 0 }
  date_of_birth:    { tier: 2, action: surrogate, budget_per_month: 25 }
  address:          { tier: 2, action: surrogate, budget_per_month: 25 }
  person_name:      { tier: 3, action: surrogate, budget_per_month: 500 }
  email:            { tier: 3, action: surrogate, budget_per_month: 500 }

The block action is worth naming because it is the one people leave out. For a tier-1 entity, replacing the string and continuing is often the wrong response — if a national identity number is present in a transcript, the safer behaviour is to refuse to send the document at all and route it to a human, because a detector that found one occurrence in a document is telling you the document is the kind of document that contains them, and its recall on the second occurrence is the same as on the first.

Where an entity-by-entity view cannot see#

Everything above treats a document as a bag of independently classifiable spans, and that model has a structural blind spot: some combinations identify a person when no member of the combination does.

Latanya Sweeney's work in the late 1990s made this concrete in a way that still governs how regulators think. Working from US census data, she reported that roughly 87% of the population could be uniquely identified by the combination of five-digit ZIP code, gender, and full date of birth — three fields none of which is an identifier and all of which appear routinely in records that had been released as anonymised. Philippe Golle's 2006 re-analysis on 2000 census data put the figure lower, around 63%, which is a meaningful correction and does not change the conclusion: a combination of ordinary attributes is very often unique.

Your detector will never flag any of those three fields, and it should not, because masking every postcode, every gender reference, and every date in a support corpus would leave documents that no longer say anything. The mitigation is not a better recogniser. It is generalisation — truncating a postcode to its first two characters, reducing a date of birth to a birth year or an age band, replacing a precise job title with a function — which is the machinery of k-anonymity and its successors, applied to the document rather than to a table. It is also considerably harder on free text than on a database column, and most production pipelines do it partially or not at all, which is an honest gap rather than a solved problem.

The second blind spot is that whether a string is personal data at all depends on the surrounding context. "Ramirez" is a common surname carrying almost no information in a transcript about a shipping delay, and the same string in a transcript about an addiction-services referral is a disclosure with Article 9 consequences. Detectors classify spans; consequence lives in the document.

The third is jurisdiction, and the two dominant regimes disagree at a level that changes engineering decisions. One of them is the GDPR, and the other is HIPAA, the US Health Insurance Portability and Accountability Act, whose Safe Harbor method offers a way to declare a data set de-identified by stripping a fixed list of eighteen identifier types out of it.

An enumerable list is enormously easier to build a detector against, which is why teams operating under both regimes tend to implement the HIPAA list and quietly assume it satisfies the other. It does not, because GDPR has no equivalent of a completed checklist — pseudonymised data remains personal data, which means the exit criterion for a European pipeline is a documented risk judgement rather than a passing test.

All of which leaves you with an axis, a scale, and a set of budgets, and no number to put into them. The next chapter is about producing that number, starting with the observation that the one your vendor puts on its datasheet is not it.