Take the ratio first, because the ratio decides whether the number transfers to you. 400ms against a 900ms direct baseline is a 44 percent penalty, and it is 31 percent of the wall clock the gateway reports [21][20]. A 900ms provider call implies short output. The post says so itself: network setup is a much larger share of total latency for short AI requests, while provider inference usually dominates longer generations [13]. If your median request streams for several seconds, the same 400ms sits in the noise. The subtraction technique still works at any request length, but the size of the alarm it raises shrinks as the request stretches out.
Averages will hide the shape of it. The author compares p50, p95 and p99 instead, because a gateway can look healthy at the median while connection setup, overloaded workers or retries wreck the tail [8]. That distinction is what makes the next step worth the effort: decompose the request into client, gateway, authentication, policy, routing, provider, streaming response and back, and stamp each boundary [9]. A gateway that reports only "request completed in 1.3s" leaves you exactly where you started [10].
Now the mechanisms, in the order they are cheap to check. If the gateway opens a fresh outbound connection per request, it pays DNS lookup, TCP setup and TLS negotiation again every time, when the provider connection should be pooled and reused, along with the connection from your application to the gateway [11]. The things to read are configuration, not code: keep-alive, whether the HTTP client actually reuses connections, pool settings, repeated DNS resolution, per-request TLS, idle timeouts that reap connections too aggressively, and whether the gateway even runs near the provider region [12].
Authentication is the next suspect, because it is synchronous by construction: request, database lookup, user lookup, permission lookup, then continue, on every call [14]. The post's illustrative figures are a 2ms cache hit against an 80ms miss [15]. That miss path is 40 times the hit path and 78ms above it [23], which is real money, and it is also only a fifth of the 400ms [22]. Even a 100 percent miss rate would not fund the whole regression. Expect a budget spread across stages rather than one villain. The fix is a matter of sequencing rather than subtraction: keep the auth check, but strip the unnecessary synchronous work around it [16].
Then logging, which is where the gap usually gets its bulk. One request can trigger request logging, token accounting, cost calculation, audit logging, trace creation, database writes, metrics and security events, and if the gateway waits for all of that before forwarding, the latency accumulates in front of the model call [17]. The whole delta between "write audit record, wait for database, continue" and "enqueue audit" is scheduling [18]. Computing what a request cost, on the critical path of that request, is an expensive way to find out.
One caveat on provenance. The author writes the connection and authentication sections in the conditional, as checks and measurements he would run [19], and the supplied text ends mid-sentence in the logging discussion without naming the stage that owned his own 400ms [25]. So the 2ms and 80ms are worked examples, not a system you can benchmark against. What you should end up with is your own stage budget in milliseconds. That artifact is also the one that tells you whether the gateway earns its place.
