Build1 publisher3 min readPublished
A 6% driver reserve decides which models fit on a $2,000 pair of P40s
One operator's surplus build serves thousands of agent requests a day on silicon that taped out in 2016. Every model on it is sized against the 45 GiB the driver actually hands back rather than the 48 GB printed on the cards.
The Engineer · Build desk
What happened
- A local inference server assembled from about $2,000 of used parts, none of the silicon newer than roughly 2016, handles thousands of agent requests a day for its operator.
- Two Tesla P40s advertise 48 GB of VRAM between them, but the driver reserves about 6% and leaves 45 GiB usable, so any budget built from the spec sheet runs out of memory.
- vLLM is off the table on these cards because compute capability 6.1 is too old, a conclusion the operator says he verified in his own experiment directory rather than accepting from the docs.
- Pascal has no Tensor Cores, so FP16 math runs at one sixty-fourth of the FP32 rate and the card's real throughput comes from INT8 through the dp4a instruction at about 47 TOPS.
- Across one measured 40-hour production window the stack processed 8,129 requests with a 0.17% failure rate and no manual interventions.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The cash payback arrives faster than the competence does: two months of avoided frontier API bills against weeks the operator says he lost discovering which optimisations these cards refuse, and that second bill lands on whoever does the tuning.
- constraint With one large model resident and roughly 30-second swaps through the Portainer API, a request for a different model waits on a load, and there is no second resident model to absorb the wait.
- exposure Guidance written for newer cards is a live failure mode rather than a missed optimisation here, since a split mode a vendor blog sold as 40% faster crashes Pascal with an illegal memory access.
- decision The 45 GiB ceiling turns context length into a model-selection decision, because the KV cache for a long window claims its share of VRAM before any weights are loaded.
Six percent of 48 GiB is 2.88 GiB, which leaves 45.12 GiB, and that is where the operator writing at dev.to gets his 45 GiB usable figure [7]. Read the same 48 as decimal gigabytes and you start from 44.70 GiB and finish at 42.02 GiB after the reserve [1]. The two readings sit about three gibibytes apart [8], which is wider than the margin the entire budgeting exercise turns on. Gibibytes versus gigabytes is a fine argument to have before the download, not after.
Whatever is left then gets shared with context. A 262k-token window costs about 23 GB in this stack with a quantized KV cache [6], which leaves roughly 22 GiB for weights [2]. That remainder is the actual model-size ceiling, and it moves every time the context configuration does.
Twelve compiled GPU stacks sit behind a single OpenAI-compatible endpoint, swapped on demand [9]. What makes that survivable with only one large model resident is that the routing does not run on the GPUs: small 4B models do classification and routing on a handful of CPU cores at 15-25 tokens/sec, always on, never competing for VRAM [10].
The throughput figures are logs from one machine, not results anyone else has reproduced. A 26B mixture-of-experts model at Q8 decodes at 41 tokens/sec [14]. The daily workhorse, a 27B dense model at Q6_K, runs 13-17 tokens/sec single-stream with speculative decoding and about 15 tokens/sec aggregate across four parallel slots [15]. For either number to transfer you need two cards of the same compute capability, the same quantization, the same speculative decoding setup, and a runtime whose INT8 kernels target 6.1. Change one of the four and you are back to a hypothesis.
The reliability figure travels better, because it is a rate rather than a configuration. The measured window works out to about 4,900 requests per 24 hours [3], with roughly 14 failures across it [4]. So "thousands of agent requests a day" [18] is arrived at without rounding up, which is not something you can say about most self-reported inference numbers.
The most portable finding here has nothing to do with Pascal. A modernized rebuild changed four variables at once and prompt processing fell from 153 to 29 tokens/sec [19], a factor of 5.3 [5], and nothing was attributable until a clean A/B separated them. The author's own framing is that on an H100 that class of mistake costs a little throughput you never notice, while on a P40 it fails loudly [20]. His closing rule is that verdicts expire and mechanisms survive [23]. On this box the mechanism is the reserve: whatever the flags do next, the driver takes its 6% before your weights land.
What to watch
- Whether later posts in the series publish the per-part eBay prices behind the single $2,000 total.
- Any independent replication of the 41 tokens/sec Q8 mixture-of-experts decode on a second pair of P40s.
- Whether a runtime with compute-6.1 INT8 kernels ever allows two large models resident at once, which would take the 30-second swap off the critical path.