Skip to content

Science1 publisher3 min readPublished

Three previews of a MacBook-sized Jev rebuild miss the release screen their author declared

kev packs a document and every typed question into one sequence and reads them in a single forward pass on a Mac. Its author scored four checkpoints against the real Jev on frozen items and published the reads that missed a pre-declared gate.

The Scientist · Science desk

Illustration accompanying Three previews of a MacBook-sized Jev rebuild miss the release screen their author declared

What happened

  • jaredpalmer/kev is a LoRA adapter plus a small readout head on Qwen base models ranging from 0.5B to 8B parameters, published with checkpoints on the Hugging Face Hub.
  • It reads a document once and answers many typed questions about it in parallel, inside a single prefill pass with no decoding step.
  • The document and all of the questions are packed into one sequence under a block-causal mask, so each question can see the document but not any sibling question.
  • On the out-of-domain suite, kev-4b scored 0.76 and kev-8b 0.77, against 0.86 for the real Jev measured on the same items.
  • Three previews ship without a version tag because each missed a pre-declared screen on held-out rule reasoning that required both siblings of a policy pair correct at 70 percent.

Compiled by The ScientistSomething wrong?How this is made

Why it matters

  • capability A team holding documents it cannot send to a vendor can exercise the System One interface on its own hardware, because the official SDK talks to a local kev server after one base_url change.
  • constraint The probabilities are only as good as the labels they were fitted to, so a queue-routing rule built on kev's confidence needs a labelled sample of the operator's own documents before it can be trusted off-distribution.
  • decision Anyone choosing between a hosted Jev call and a local pass is now choosing between measured accuracy on unseen sources and keeping the inference in-house, and the deciding input is how far their documents sit from kev's training set.
  • precedent A pass/fail screen declared before the run and published alongside the checkpoints that missed it gives reviewers a threshold to hold the next small-model release to.

A trained layer produces the number. A pointer head scores each question's options against its decision token, applies a softmax, and those probabilities are the answer [4]. The head learned them with cross-entropy against labelled outcomes, which the README describes as probabilities "learned rather than generated as text" [5]. Three question types go through one shared readout: yes/no, a choice of 2 to 255 options, and ordered score levels [7]. The design follows a published reconstruction of TypeSafe's Jev, "Jev's Architecture Unmasked" [8].

Training against outcomes fixes the shape of the output. Whether the numbers hold on a document unlike the training set is a separate question. On the out-of-domain suite, kev-8b posts a Brier score of 0.34 and 8 percent confident errors on sources it never trained on [10]. That rate needs a denominator. If it is the 764 out-of-domain records, about 61 came back confident and wrong [26], and an auto-escalation rule would have acted on all of them.

Pack many questions into one sequence and one specific failure comes with it: a later question conditions on an earlier one's wording. The block-causal mask is the guard [3]. The evidence it worked is that packed requests and separate requests agree to 4e-6 [6]. That agreement is what rules out sibling leakage as an explanation for the parallel scores.

The table's rows all sit on the same frozen, checksummed items: 1,200 in-distribution questions from the trained sources, and 764 out-of-domain records from six public sources plus held-out programmatic policy rules [13], for 1,964 items in total [22]. Doubling from 4B to 8B parameters moved the out-of-domain score by 0.01 [24]. Jev sits 0.09 above kev-8b [23]. The README gives these as bare numbers and does not name the metric.

The items are matched on difficulty. Serving kev-4b costs a 32 GB Mac running bf16 [16]; a Jev call costs whatever TypeSafe charges for it.

The release screen is the part I would want other small-model releases to copy. The gate was set in advance: both siblings of a policy pair correct at 70 percent or better on held-out rule reasoning. The best preview read 0.67, three points short [14][25]. Each model card records one locked-test read, so no best-of-several selection sits inside the figure [14].

Serving is where the laptop comes in. kev-0.5b trains in about an hour and three quarters on an Apple M5 [15], while the 4B and 8B recipes take 40 to 70 minutes on a single H100 rented through Modal before they serve on the Mac [16]. Serving is tested on Apple Silicon; training and evaluation ran on CUDA and MPS [18]. The README points new users at kev-4b, "the best accuracy per byte" [17]. In its own worked example, that checkpoint answered a choice, a yes/no and a score question about one customer complaint in 277 milliseconds on an M5 [19], and the yes/no on whether the case needed urgent human attention came back at 0.54 [27].

What to watch

  • Whether a version-tagged release clears the 70 percent policy-pair gate, and on which checkpoint size.
  • Whether anyone reproduces the 0.77 out-of-domain read on their own documents and their own labels, since the published suite is frozen.
  • Whether TypeSafe's System One request and response shapes stay stable enough that a local kev server keeps answering the official SDK.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories