Providence's 2-billion-note de-identification run had to clear three separate tests

A recall figure only means something against the baseline it displaces. In the MIT study of nursing notes Talby cites, individual clinician reviewers caught between 63% and 94% of identifiers, two working together averaged 94%, and three together still left PHI in place ^[8]. Manual review remains the gold standard in most organizations, and Talby's argument is that it does not deserve the title, because reviewers tire, skim and disagree ^[9]. Set against that, the reported 98.6% sits above the ceiling three humans reached jointly ^[7][8].

The labeled set held 16,246 identifiers and the system returned 16,020, so 226 went through ^[20]. Talby's own caveat is the one worth holding onto: accuracy gives you the size of the residual and nothing about its shape, and 226 misses spread thinly across a corpus is a different problem from 226 concentrated on a handful of patients ^[10].

Which is why the second test is the one buyers have least intuition for. The 2023 MIT-led audit of nine commercial and open-source tools found statistically significant gaps across demographic groups, with rare white names caught more reliably on average than common Asian ones, and all three major cloud services scoring worse on Black and Asian names than on white and Hispanic ones ^[11]. The two hardest names in that audit were "An Dizon" and "An Son", because the models read "An" as an English article ^[12]. A pipeline with that profile applies its weaker protection to the patients whose names it handles worst, and Section 1557 of the Affordable Care Act bars discrimination on race and national origin in federally funded health programs, with an updated HHS rule published in May 2024 ^[13]. Providence's answer was to sample 24 subgroups defined by sex, state and race, sized so small populations still produced reliable estimates, and to report differences inside statistical noise everywhere, including on patient names ^[14]. Earlier bias work used small synthetic datasets; the authors believe this is the first equity evaluation of medical text de-identification on real-world data at scale ^[15].

The skeptic's objection is sitting on the byline. Talby is CEO of John Snow Labs, and this account of his own system's validation ran as a Forbes Tech Council column ^[1][21]. That is a reason to treat the scores as claims awaiting replication and the test design as the part that transfers. The equity result is the most exposed of the three, precisely because an independent audit already found gaps in nine other tools ^[11], so a second real-world corpus is the check that would settle it.

The third test is where the available record runs out. Re-identification in practice looks like the 1997 case Talby recounts: a graduate student paid $20 for Cambridge, Massachusetts voter rolls, joined them to anonymized state employee health records on birth date, sex and ZIP code, found six people sharing the governor's birthday, three of them men, one in his ZIP code, and had his medical file ^[17]. The same work estimated that 87% of Americans were uniquely identified ^[18]. The account sets up a red team given three months and no rules ^[16], and does not carry what the team found ^[22].

Nor does it address the failure mode that arrives later. The three tests answer missed identifiers, uneven protection and adversarial joining; none of them is a schedule for re-testing when note templates change, new clinicians arrive, or the patient mix moves ^[23]. Expert Determination is a judgement about a corpus at a moment ^[3]. A buyer who accepts a vendor certificate this quarter still owes an answer next quarter on which patients the pipeline protects least well, and the certificate will not contain it.