Inference latency engineering

A 7B model at 11 tokens per second cleared the bar for private contract Q&A

Single-user inference streams weights out of memory, so the sizing question for a private document assistant is RAM and prompt length rather than which accelerator a vendor quoted. The laptop in question was three years old.