Ninety Milliseconds Of Work Inside A Ten-Second Request

The arithmetic in that single trace is the entire investigation. Application processing came to roughly 70ms, the database to about 20ms, external calls looked normal, and the client waited nearly ten seconds ^[7]. So about 90ms of accounted work sat inside a 10,000ms request: something close to 0.9 percent of the wall clock had any instrumentation pointed at it, and roughly 9.9 seconds did not ^[18].

Every theory in the room before the trace was a theory about that 0.9 percent ^[5]. An index makes the 20ms smaller. A garbage collection tuning pass works on the 70ms. More application logs describe the 70ms in greater detail. None of them reach the interval where the request was actually sitting, which is why the first pass through recent commits, queries, serialization, background jobs, caching and new endpoints found nothing suspicious ^[4].

What it was: under certain traffic patterns connections were not returned to the pool quickly enough, the pool occasionally exhausted, and new requests waited for one to free up ^[8]. That failure mode is self-clearing, which is why it presented as something periodic and mysterious that recovered without anyone touching it ^[2]. It is also why the GC guess was reasonable in shape and wrong in location. Both look like a slowdown on a timer.

The number that separates the two problems is the gap between execution and admission. A query taking 15ms can sit inside a request taking ten seconds ^[9], which is about 667 times more time spent waiting to start than doing the work ^[19]. From the user's chair that is indistinguishable from a slow query, and from the engineering side the two share no code path.

The repair was a connection lifecycle correction, a pool configuration change, and a new metric on connection acquisition time ^[10]. Before that, the only database timing collected was query execution ^[11]. The author's claim is that the acquisition metric would have exposed the problem almost immediately ^[12], and the structure of the trace supports it: an entire queue stage existed with no clock on it.

Worth noting what the write-up does not contain. It gives the baseline of 80-120ms and the incident range of 8-12s, a hundredfold degradation at both ends of the band ^[3][17], but it publishes no post-fix latency distribution, so the recovery is asserted rather than shown. The generalisation about who solves this is the author's own: in interviews, junior engineers name technologies while senior ones ask where the latency lives, whether it hits every endpoint, whether it tracks traffic, and whether some pool of connections, threads, sockets or workers is being exhausted ^[13][14]. Treat that as one practitioner's hiring heuristic, not evidence.

The transferable part is narrower than "add observability". A machine can be nearly idle while its users experience terrible latency, because the system is not computing, it is waiting ^[15][16]. Almost every dashboard measures work. The interesting seconds accumulate at the handoffs where work has not started yet, and those need their own clocks.