Several interview transcripts went through the first version of the author's Interview Coach and the results looked pretty good, which is exactly when she started asking whether it was good enough to roll out to all of her students [7]. That is where a rollout stalls. A handful of good-looking outputs is a sample, and reasoning from a sample is a research job before it is an engineering one.
The mechanism is easiest to see in the guide's arithmetic. For a function that adds two numbers, you can write assertions that 2 and 3 return 5, that 3.2 and 4.1 return 7.3, that -1 and -3 return -4, and that 2 and 3.1 return 5.1, then run them again every time the code changes [10]. That is four cases with one expected value each [13]. Give the same job to an LLM and the expected-value column stops being a number, because the same input can produce a different answer and because semantic tasks often have several answers of varying quality [5]. What fills that column is a written description of an acceptable answer. The person who owns descriptions of acceptable answers is the person who wrote the requirement, not the person who implemented it [8].
Product Talk states flatly that evals are the only way to know whether our AI products and workflows are any good [11]. That is the author's working position, not a measured result. The narrower version is more useful: the customer-facing question the guide poses is how you know the thing is consistently good across all of your customers and all target use cases [16]. That is a coverage question, and coverage is settled by whoever chose the sample.
The internal case will land first, because it is already happening on most product teams. The guide names four everyday jobs where PMs lean on AI, including writing PRDs, synthesizing customer feedback and interviews, analyzing behavioral analytics, and making sense of meeting notes [12][14]. Its checks against those jobs are pleasingly unglamorous: whether the generated PRD includes everything that was asked for, whether any of the customer quotes are fabricated, and whether a long calculation started from the right question and stayed there [9]. All three name the failure before they measure anything.
That points to a simple test worth running on any AI output your team ships or circulates: place it on two axes, asking whether you can write the failure you fear in one sentence that points at something observable in the output, and whether that output leaves your team. When the failure is nameable and the output leaves the team, the team can write the check itself, since it is usually a lookup against a source document rather than a model. When the failure is nameable but stays internal, running the check once shows whether the failure is real before anyone builds a guard for it. When the failure is unnameable and the output leaves the team, that is a requirements problem wearing an eval costume, and no test suite will discover the acceptance criteria on its own. When the failure is unnameable and internal, someone still has to read the output themselves. That is roughly the shape of the eval as it stands today.
