A Gulf bank's compliance rule priced out to $133 of GPU per seat

A residency rule converts into a VRAM budget, and that is where this build gets interesting. The rule of thumb in the write-up is roughly as much VRAM as the model weights, putting a 7-8B model at 4-bit quantisation on a single 16 GB card such as an RTX 4090 or an A10 ^[6], and a 70B on two or four 24 GB cards depending on quantisation ^[7]. The two 24 GB 3090s in the bank's stack come to 48 GB, which is exactly the floor quoted for that 70B tier ^[14]. They ran a 13B on it for about thirty users ^[20]. So the ceiling on answer quality here was a latency and concurrency decision, not a silicon one, and the author is explicit that buying up to A6000-class hardware for frontier reasoning means asking whether the quality gap justifies four times the cost ^[3].

Divide the spend by the seats and the capital argument thins out further: under four thousand dollars across roughly thirty users is about $133 of hardware per seat, one time ^[21]. Worth noting that the account is internally inconsistent about what that money bought. The server room is described as already holding two half-idle GPU boxes running dashboards and a transcoding job ^[18], while the shipped spec is listed as two used 3090s bought cheap ^[20], and those are different starting positions for anyone budgeting from zero ^[15].

The software substitutions are where the API removal actually shows. Embeddings come off the GPU entirely: bge-m3 or e5 on an eight-core CPU will chew through a 1,000-page corpus overnight and a few hundred chunks per second at query time ^[1]. The vectors go into pgvector, Qdrant, LanceDB or Milvus that you operate ^[8], and generation is served locally through vLLM or Ollama ^[9]. The author's position on frameworks is that they create a second source of truth for prompts and chunking that you end up owning anyway, and recommends FastAPI, an ingestion scheduler, and functions for embed, retrieve and generate ^[2]. Defaults given are 700-token chunks with 100 tokens of overlap ^[10].

The failure mode is the part that has no hardware fix. The bank's compliance team edits documents constantly, and an index that is not idempotent and versioned on ingest will quietly answer from retired policy ^[11]. That was the risk the project was commissioned to remove in the first place ^[11], and once the hosted endpoint is gone there is no vendor sharing the consequence of a stale retrieval.

One caveat on all of it: this is a single first-person build account published on dev.to ^[13], seven weeks from start to live ^[19], with no independent confirmation of the spend or the user count. The parts that generalise are the ordering. Decide whether generation must be local before buying anything, because self-hosting retrieval while calling a hosted LLM still sends the queries out ^[5], and that configuration fails the stated policy ^[16].