A note checker with no accuracy figure, and the labelled dataset it borrowed to show its misses

The two failures in this report do not behave alike, and that difference is the reason to read a disagreement table rather than a score.

The pertinent-negative rule is code with no model in it, and it failed identically on every note it touched. It located the doctor's question about blood clots, migraines with aura and uncontrolled blood pressure, and never registered the patient's "Correct, none of those" as the answer ^[10]. That class of error is cheap: enumerable and reproducible, so it can be fixed. Less comfortable is the precision underneath it. Only two flags in the whole run were correct, both on a weight change recorded as negative when it was asked about and never answered ^[11]. The arithmetic also does not close. Twelve flags are the denial bug, a thirteenth misread a phrase about foot swelling, two were right, which leaves one of the sixteen never described ^[18].

The other failure has no patch. In dialogue_2 the visit is history only, and two notes assert the cough is "likely related to allergic etiology"; all three of Omi's judges counted that a major unsupported claim and Krasyn's judge called it Supported ^[12]. The shape recurs three more times in the sample, including "Type 2 diabetes mellitus" written where the doctor said "your diabetes" or nothing at all ^[13]. Krasyn's explanation is that its judge prompt allows faithful clinical translation to count as Supported, and it adds the line that matters most: each of these reads as sensible, which is why the signing clinician would not catch them either ^[14]. Across all eighteen labelled notes, the tool applied the Unsupported label exactly once ^[19].

Against that, a headline percentage would have been a courtesy to the vendor. No clinician-adjudicated reference set exists to compute one from ^[2], so any figure would have been Krasyn grading itself, and the labels it borrowed instead are Omi's, published under MIT with the dialogues and the model-written notes ^[4].

The exercise is small and half-blind, and says so. Six synthetic dialogues, short and clean, with LLM judges on both sides and no clinician adjudication on either; the column that says which side looks right belongs to the vendor ^[16]. Only half the sample is comparable at all, because the 2026 benchmark ships per-writer totals rather than per-note counts ^[5], leaving eighteen of the thirty-six pairs with nothing to disagree with ^[20]. And the checker is not fully stable against itself. Verdicts held on seven of eight pairs run twice, while the omission list changed on four of eight, which is why that list is offered as a prompt to look rather than a count ^[15].

None of this makes the report weak. It makes it readable by someone deciding what to buy, because both error classes are visible separately instead of averaged into a single number that hides which one would reach their patients.