Serialized queueing is the only reading of the latency that also explains the memory. Sessions, the FastAPI app and the model all lived in one process, so every queued conversation added to the resident set until it went past 30 GB [7][4]. The queue was simply consuming memory as it grew.
The measured jump is roughly 39x [21], and strict one-at-a-time service of forty turns overshoots the observed figure by about a second [20]. So nothing in the model needs to have got slower. One slow inference blocked every request on the same code path [7], and there was no ceiling on what could pile up behind it [3].
The HTTP 504 response came from the gateway, not the application itself [5]. Nothing in the path had been told what to do when demand exceeded capacity, which is the property the guide's author, writing on dev.to, calls backpressure: queue, reject politely, or shed load, and never silently drop [10]. When one worker fell over it took the whole process with it [6], because there was no boundary between the layers to contain it [9].
Treat 47 seconds as a claim about that stack. It transfers to yours if three things hold: model serving, API logic and session state share a process [7]; the queue in front of inference has no bound [3]; and conversation state sits in process memory rather than an external store [11]. Break one of the three and the curve bends instead of snapping. That is why the author, who says he has had this conversation about two dozen times in two years [18], calls the fix architecture rather than hardware, and warns that buying capacity for a prototype gets you a six-figure cloud bill and a service that still collapses [8].
The cheap fixes belong in week one. The service layer gets an explicit input budget and a hard timeout on the model call, which he names as the two details most people skip [17]. Conversation IDs map to Redis keys instead of process memory, which is what makes it safe to run twenty replicas and kill any of them mid-flight [16].
The residency rule removed the easy option here. Payment and inventory data could not leave the VPC, so they self-hosted with vLLM [13], against his stated 2026 default of a hosted endpoint unless the workload is high-volume or data-sensitive [12]. Hence the one thing he insists on either way: put the model behind a single interface you can flip with an environment variable, written in week one, because the day you want to flip it is not the day to be editing call sites [14].
The metric that would have shown all of this before the load test is queue depth, which he puts on the day-one dashboard next to latency percentiles, GPU utilisation and token spend [25].
