Producttalk.org's eval harness turns a 120-check comparison into three commands

One of the transcripts in the repo describes a participant who opens the app every single morning and has already decided to cancel at the end of the month. The baseline prompt gets her status wrong ^[5]. The story it writes about her reads fine. You only catch the error because a golden dataset eval holds a label the story disagrees with ^[6].

Most teams say they agree on what quality means before they measure it. In practice, they open four or five outputs, find one bad quote, edit the prompt, and move on. That is not laziness. A real baseline means running every input through the prompt, saving each output, running every eval against every output, tallying the results, and then repeating the lot for each variant, which producttalk.org says you can do by hand until it gets tedious ^[11].

Count the tedium. Fifteen transcripts through two prompts is 30 stories ^[15]. Thirty stories against four evals is 120 scored checks for one before-and-after ^[16]. In the repo, that is three commands, because each of the three steps is a single one ^[19].

The reported effect of adding grounding rules and status definitions to the prompt: fabricated quotes went from 12.5% to 1.7% ^[8], a relative drop of about 86% ^[17], and ungrounded claims went from 8.4% to 3.2% ^[9]. The second number matters more, because 3.2% still works out to roughly one claim in every 31 ^[18]. Whether that residue is trivial or the kind that ends up quoted back at you in a readout is not something the summary table answers, which is why the harness also writes a failures file per eval in prose a person can read ^[12].

Order your own checks by what each one costs to trust. Two of the four here are code assertions: quotes must appear word for word, and the model must produce the number of quotes the prompt asked for ^[6]. Those need no labeling and no faith. The status eval needs a correct answer for each transcript. The fact-checking judge needs the most, and it arrives with a calibration set of 29 hand-labeled claims precisely because the judge is an LLM too ^[6]. Teams reach for the judge first, since it feels closest to measuring quality. It is also the only one of the four whose own output you have to audit.

The honest caveat is that the transcripts are synthetic ^[4], so every trap in them is one the author knew to set: the sister paying for the annual plan, the roommate with 20 plants ^[4]. Real inputs will surface failure modes the harness's author never anticipated. That is an argument for cheap re-runs, not against the example.

So the forcing function, before anyone in the room argues about which metric matters. Two artifacts on the table: a number attached to a fixed input set, and a failures file that someone who was not in the room can read. Then the rule producttalk.org applies to its own experiment, which is that a change fixing one error while making another worse is not an improvement ^[13].