Skip to content

Topic

Pre-registered benchmarks

Benchmarking discipline in which models, data, metrics and comparisons are committed publicly before any results are collected, with deviations recorded.

Current clusters

build1 publisher

A 4-bit Gemma 4 26B on one L4 trails TypeSafe's Jev by 2.1 points overall

A pre-registered run reads Gemma's label logits on one 24 GB GPU and scores it against the hosted API on the same 3,880 records, where it is level on yes/no questions and 4.5 points behind on multiple choice. One temperature fitted on 50 labels closes the calibration gap.

Publishers:dev.to

Reality

Evidence70
Adoption20
Hype gap−12
Incentives35
Confidence55