Averaging IoU across classes buries the failure you commissioned the check to find

IoU is overlap area over combined area, so the dev.to example's 70 units of overlap inside a 100-unit union gives 0.70, and the tight pair's 95 units gives 0.95 ^[1][2]. Invert it and the number gets useful: 30 of 100 union units disagree at 0.70, 5 disagree at 0.95, so the tight box cuts disagreeing area by a factor of six ^[3]. Which of those is acceptable is a property of the consumer, not of the annotation ^[3]. The post's own contrast holds. A shelf-counting model absorbs positional slop that would be a defect in lane detection, where a few pixels of boundary drift is not rounding error ^[4].

Masks have the same problem one level up. Segmentation is scored as mean IoU across classes, and a mask can be excellent on the large easy classes while failing on small or visually ambiguous ones, which the average absorbs ^[5]. Dice, the close cousin that weights the overlap differently, turns up in medical and fine-grained segmentation where the boundary carries the meaning ^[6]. Neither metric tells you which class went wrong unless you break it out.

The span example is where a passing score hides the finding. Across 200 transcripts the gold set holds 340 true product mentions; the annotator tagged 310 spans, 290 of which match, giving precision of about 0.94 and recall of about 0.85 ^[10]. F1 is their harmonic mean, which for counts is 2 x 290 / (310 + 340) = 580 / 650, or about 0.89 ^[1]. On an acceptance sheet, 0.89 passes. Underneath it sit 50 real mentions that were never tagged and 20 spans tagged over nothing, so misses outnumber false positives by two and a half to one ^[2]. The author reads that skew as guideline coverage rather than carelessness: variant product names and abbreviations the instructions never enumerated ^[11]. Flat accuracy cannot make the distinction, because it does not separate a miss from an invention from a boundary that landed on "York City" instead of "New York City" ^[9][8].

Two mechanical details are worth copying outright. Points have no area, so IoU does not apply; the check is the distance between the annotated point and its true location, normalized against something measured in the same image, since raw pixel distances are not comparable across shots at different scales ^[7]. And every metric above can pass on a dataset that is missing labels entirely, so completeness is a separate pass: compare object counts per image against the expected distribution, then open the images reporting zero when most should report ten to thirty ^[12][13].

These figures are worked examples inside an explainer, not measurements of a real delivery ^[15]. For the method to transfer, the gold subset has to be drawn the same way as the batch it judges, and the downstream tolerance has to be known before the threshold is set. That is what turns the exercise into a decision.

The part of this with no threshold at all is disagreement triage. Where two annotators diverged, the post sorts the cases into a guideline that did not anticipate the example, one annotator missing an edge-case rule the other applied, and a borderline judgment where both readings are defensible, and it assigns each a different fix ^[14]. The third is the expensive one. When both readings are defensible, the label set itself is the defect, and only a guideline rewrite fixes it.