The redaction figures are more useful than their size suggests. Anthropic removed or altered 1.9% of Stanford's clusters, and those clusters accounted for 4.28% of Stanford's conversation sample [8][9]. Divide one by the other and the cut clusters were roughly 2.3 times the average size [1]. Oxford's equivalent ratio is close to even at 1.16, METR's is 1.64 [2]. Across the three samples the withheld material covers about 27,700 conversations, near 3.7% of the pilot's 750,000 [3]. Anthropic says most removals concerned descriptions that could reveal how users bypassed safeguards [10]. That is a defensible reason to cut, and it is also the category an outside evaluator would most want to read.
The interface determines what is answerable at all. Researchers write questions called facets, Anthropic runs them, Claude classifies and clusters the conversations, and the outside group receives category counts, percentages and cluster prose [5][6]. Nobody outside the lab ran an unrestricted search or saw a raw conversation [5]. So every number in METR's Claude Code work is the output of a classifier owned by the company whose product is under study, applied to data the evaluator cannot inspect. There is no move available inside this arrangement that lets METR pull two hundred sessions by hand and check whether a facet was scored the way it was meant.
Anthropic's contractual review covered privacy, information that could help users evade safeguards, confidential company information, and research accuracy, with the outside groups keeping control of their questions and conclusions [7]. Three of those four are unremarkable. Research accuracy is the one that matters, because it seats the subject of the study in the loop on whether findings about it are correct.
The sampling frame sets a second limit. The Claude.ai samples came from Free, Pro and Max users, with Team, Enterprise and API traffic excluded, while METR's Claude Code sample was consumer users who had opted into data use for model improvement [4]. Stanford's headline result, that 56% of conversations with an actionable task involved consequential or higher work and 12% reached its high-stakes tier, concentrated in legal and financial guidance [13], is therefore a statement about individual subscribers. It says nothing measured about the deployments that carry contracts and administrators.
Imperial College London's red team could not reidentify users from the released material, but it did link one cluster to a widely used open-source project from distinctive language associated with the tool [11]. Anthropic's stated remedy is higher minimum-user thresholds and less distinctive cluster descriptions [12]. Both raise the floor on what an outsider can resolve, which is the trade being made: privacy margin bought with descriptive detail.
None of this reads as bad faith. It is a question of what aggregate counts can bear. They can describe a distribution, including the finding that 72% of conversations were human-led and friction appeared in 49.7% [14][15]. They cannot support an audit, because auditing means going to look at the specific thing you doubt, and that is the one operation the pipeline does not offer.
