Caching the prefix stops a 0.5B model re-encoding the same instruction block 600 times

Narrow automation prompts are mostly fixed text: a task instruction, a taxonomy definition, and a few examples, with a short tail that changes per item ^[6]. The article's illustration is an instruction block of a couple of hundred tokens and twenty or thirty tokens added per ticket ^[7]. Take 200 and 25, and about 89 percent of every prompt is identical to the last one ^[14]. Over the 600-record dataset, re-encoding in full pushes 120,000 prefix-token positions through every layer, against 200 if the keys and values are computed once and held, so 119,800 of those encodings are avoidable ^[15]. Token counts alone show the redundancy; ranking prefix reuse against other tuning choices needs seconds on a clock, and the text available ends inside the baseline script, before any timing for the cached path ^[18].

Copying the script as published gets the four-byte path, which at half a billion parameters is roughly 2 GB of weights instead of roughly 1 GB ^[16]. That is a different starting point for any prefill figure a reader measures at home.

The script sets the thread count to the number of CPUs ^[12], so what is being timed is single-stream inference on a laptop. On a GPU server taking concurrent requests, I would expect the prefill share to be smaller, because batching spreads one shared prefix across several items and the held keys and values occupy memory that the weights and the in-flight batch also want.

Reuse holds only while the prefix tokenizes identically on every call. The walkthrough writes the ChatML layout out by hand instead of calling apply_chat_template(), so the prompt can be split at a known token boundary ^[10]. The scoring trick carried over from the first article has its own condition: the labels' first tokens have to differ, and the code stops if they do not, with the message "Labels share a first token; score full label sequences instead." ^[11]