Build1 publisher2 min readPublished
Kaitchup traces Bonsai 2's 98.2% retention figure to unpacked weights on an H100
Ternary weights at 1.76 bits put a 27B model into a 5.9GB file and let a laptop decode it at 28.1 tokens a second. The retention figure comes from Prism ML's own benchmark suite, not the table on the model card.
The Engineer · Build desk
What happened
- Prism ML's Ternary Bonsai 2 27B stores each weight as minus one, zero or plus one with a scale per 128 values, averaging 1.76 bits per weight and producing a 5.9GB file.
- The vendor reports an aggregate 83.9 against the full model's 85.4, the 98.2% it advertises, measured on Prism ML's own benchmark suite; the table published on the Hugging Face model page is a separate set of numbers.
- On llama-bench at batch size 1 an RTX 5090 decoded 129.9 tokens a second against the H100's 113.9, while the older A100 managed 73.9, behind a consumer RTX 4090.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The memory saving and the integration cost land on different budgets: whoever maintains the serving stack absorbs a runtime migration before the 5.9GB file ever loads.
- decision For single-stream local inference, paid datacenter GPU hours now buy less decode throughput than a desktop consumer card, so renting that capacity pays for capability this job cannot use.
- constraint The 262K window is comfortable on a laptop when context accumulates through a conversation; pasting a document of that size in one shot means waiting about eleven minutes for the first answer.
- contradiction Prism ML's published table covers the categories that survive compression best, while Kaitchup says the agentic coding runs, where the gap should be widest, were made and left out.
Decode reads every weight once per token, so the ceiling is memory bandwidth divided by file size. The dev.to write-up puts the Apple M5 Pro at about 204 GB/s [13]. That sets a ceiling of 34.6 tokens a second for a 5.9GB file, and llama-bench measured 28.1, about 81% of it [1]. The FP16 original is 54GB, too large for that laptop to load; the same arithmetic would put it near 3.8 tokens a second [14].
27 billion weights at 1.76 bits is 5.94GB [2], so the download is the parameter count times the advertised density [2].
The account, a dev.to post by Nokka written with an LLM through Nous Research's Hermes Agent and edited by Nokka [19], says the 9.3x smaller file runs about nine times faster on the same hardware [20]. Only one side of that ratio was measured. 28.1 against 3.8 is 7.4x [6], and the 3.8 is a bandwidth estimate [14].
On retention, the announcement's suite and the model card's table are two different measurements [4]. For 98.2% to describe a local deployment, the ternary file under Prism ML's runtime would have to reproduce what the unpacked weights scored on the H100, and your outputs would have to stay inside the 82K response cap those runs used [9]. Kaitchup says the numbers measure the model's accuracy; the compressed stack is not what they measure [9].
Per category, math retained 99.5% and vision 96.3% [3], from vendor scores of 96.57 against 97.06 and 78.59 against 81.64 [5]. Instruction following came out above the original, 82.66 to 81.25 [5]. Kaitchup, which tested the first Bonsai, says the headline figure is a ratio of average scores, and that long multi-step jobs such as coding agents fail first under compression because small errors accumulate [7].
Bonsai 2 will not load under stock llama.cpp: it pairs an activation conversion with the weights that the upstream runtime does not support, so Prism ML ships its own [10]. Two files are published, PTQ1_0 at 5.9GB and PQ2_0 at 7.2GB [11]. The model thinks before it answers by default, in xhigh mode, and a thousand thinking tokens on the M5 Pro is close to forty seconds before the first visible output; the documentation recommends medium [17].
Prompt processing on that laptop runs 387 tokens a second [16]. Filling the advertised 262K context [6] is 677 seconds of it, about eleven minutes [5], against roughly a minute on an RTX 5090 [16]. The model card reports more than 2.5 million downloads in the first five days [18].
What to watch
- Whether Prism ML publishes the agentic coding numbers it ran, and whether those runs use the ternary file or the unpacked weights.
- Whether llama.cpp upstream adds the paired activation conversion, which would remove the runtime port from the adoption cost.
- A third-party accuracy run on the PTQ1_0 and PQ2_0 files themselves, on consumer hardware, with a response cap above 82K tokens.