The 20-30% nobody prints: what "83% of human consistency" is a share of

Read the figure carefully and it is a ratio, not an accuracy. The agents scored 83, 82 and 86 percent *of* the participants' own two-week test-retest consistency, with demographics-only agents at 74 percent ^[2][3]. That benchmark is a person answering the same General Social Survey items twice, two weeks apart, and disagreeing with themselves some of the time ^[7]. Divide by it and you have normalised away the part practitioners most want to know: how much of the remaining gap is agent error and how much is the instrument.

The interesting number is the one left over. Against that human ceiling, the best configuration still misses 14 percent, and interview-only or survey-only agents miss 17 and 18 percent ^[6]. Two hours of semi-structured interview per person, on 1,052 people, buys you that ^[1]. And the authors say combining interviews with surveys produced only modest gains over either source alone, which they interpret as predictive benefit asymptoting once enough within-domain evidence has been seen ^[4]. If more data mostly stops helping, the residual is where persona fidelity lives, and it is not obviously shrinkable by collecting more.

The subgroup claim is worth reading as written. The paper reports reduced accuracy disparities across racial and ideological groups relative to demographics-only agents ^[5]. Reduced relative to a baseline that had only demographics to work with is a low bar to clear, and the paper as abstracted does not put a floor under per-group accuracy. A niche group is exactly where a percentage-of-consistency ratio hides trouble, because its own test-retest denominator is estimated from few people.

Then there is what happens when the output reaches a human. In the innovation-screening experiment reported by CIO, 228 evaluators assessed nearly 50 MIT challenge submissions and accepted LLM recommendations 67 percent of the time, agreeing with black-box and narrative recommendations around 75 percent of the time but with human decisions only 54 percent ^[8]. Narrative explanations cut false positives and substantially raised false negatives, because, the researchers argue, a persuasive rationale suppresses independent verification ^[9]. A simulated panel that is 86 percent of the way to human self-consistency, delivered with a fluent explanation, is a tool designed to be over-trusted.