A 4-bit Gemma 4 26B on one L4 trails TypeSafe's Jev by 2.1 points overall

Reading the labels out takes a few lines of client code. End the prompt where the answer starts, take the scores of the allowed label tokens, and softmax over those ^[8]. One forward pass, no generated text ^[6]. TypeSafe's Jev sells that same operation as a hosted service ^[7].

Whether the 2.1-point gap is your gap depends on what you ask. Gemma is level with Jev on yes/no and 4.5 points behind on multiple choice ^[2][3]. Take the suite as those two buckets: an overall gap of 2.1 needs multiple choice to be roughly 47 percent of the 3,880 records, since 2.1 divided by 4.5 is 0.47 ^[23]. Traffic that is mostly yes/no routing should expect a gap near zero. A five-option grading workload should budget for the 4.5 ^[3].

Calibration parity is not free either. It costs one temperature fitted on 50 labels from your own task, and that brings Gemma's median calibration error within 0.01 of Jev's ^[4]. Nobody had published an accuracy or calibration figure for plain Gemma read this way before this run ^[20].

The card sets the rest of the design. An L4 has 24 GB, so both 26B arms run at 4 bits ^[13]. Twenty-six billion parameters at half a byte each is about 13 GB of weights, which leaves roughly 11 GB for KV cache and activations ^[24]. Both checkpoints come from the same uploader with the same quantization settings, so the model and the read are what differ ^[14]. Both arms share one set of serving flags, the plain arm builds its prompt from the served model's own chat template ^[15][22], and the instance is a single g6.xlarge in us-east-1 running a vLLM nightly that contains PR #57250, one model at a time ^[17]. The security group opens port 8000 to one address ^[18]. These are 4-bit AWQ numbers, so a bf16 deployment on a larger card is a different measurement ^[16].

Google's Gemma account said on September 18 that DiffusionGemma read through that PR was "yielding well-calibrated decision distributions", and no measurement was published behind the description ^[19]. This run measures it on the same GPU and the same scoring code: accuracy level with the plain read, calibration better before fitting and similar after ^[5].

The process is the part worth copying. Models, image, serving flags, data, metrics and comparisons went into PREREGISTRATION.md before any model call ^[10]. The latency and small-model arms arrived in one addendum and the public-suite run in another, each committed before its own first call, with every deviation from the plan recorded ^[11]. The pre-registration also commits to publishing every result, including any where either arm does worse ^[12]; benchmark tables that promise that in advance are rare enough to be worth noting. Every per-item output is committed, on records where Bespoke Labs has already published Jev's results ^[9][21].