Team splits one 100-thread pool into database and API executors after p99 latency hit 18 seconds
Spring Boot p99 latency climbed from 400 ms to 18 s at about 20% CPU when one 100-thread pool ran both API and database calls, a DEV Community post says. The team's fix was a separate executor for each resource, and the post says virtual threads would not have helped.
The Engineer · Build desk
Key figures: the database executor has 18 threads, set as Hikari's maximum pool size of 20 minus 2 for request-path queries. The Hikari pool has 20 connections. The API executor has 80 threads, annotated as about 200 calls/s x 0.3 s p95 = 60, plus headroom.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Database executor threads (Hikari max 20 minus 2) | 18 threads | 7 | |
| Hikari pool connections | 20 connections | 2 | |
| API executor threads | 80 threads | Code note: ~200 calls/s x 0.3 s = 60 | 8 |
What happened
- Thread dumps lined up with Hikari metrics showed nearly every pool thread blocked, all 20 connections in use and up to 60 threads waiting for one.
- In one code path, the external API call and the write of its result both ran inside an open transaction ahead of the commit, so its connection stayed checked out for the whole API call.
- The database executor got 18 threads: Hikari's maximum pool size of 20 minus 2 for request-path queries that use the same pool.
Why it matters
- Who's exposed exposure Unless that transaction path was changed, a slow API response still takes a Hikari connection out of the pool while no query is running.
- What it limits constraint Raising Hikari's maximum pool size now means resizing the database executor in the same change, because the 18 is derived from it.
- What it costs cost The split needs no added threads, 98 in total against 100 before, but it leaves two executors to monitor where there was one.
Low CPU is what a blocked pool looks like. Blocked threads use almost none [4], so a service can sit idle on the processor while its p99 reaches 45 times the 400 ms baseline [15].
The two waits fed each other. Slow external calls held their threads longer and left fewer for other stages, including database work, which sat in the executor's queue [11]. Threads waiting for a Hikari connection were just as unavailable as threads waiting on HTTP, and API stages queued behind them [12]. When 60 of the 100 threads were waiting in Hikari, at most 40 could be doing anything else [16].
Past the connection count, extra database threads have nothing to do except wait for a connection to free up [9]. Once the executor is aligned to the pool, the excess work sits in the executor queue instead, holding no thread and no connection [10].
The post's summary says virtual threads would not have helped [21]. We think the sizing note above is the reason. A thread that needs the database first asks Hikari for a connection, and if none is free it waits [13]. A virtual thread would face the same 20 connections. A transaction that holds a connection across an API call would pin it whatever runs the thread [5].
The 80-thread figure comes from Little's Law, which the post applies to the API executor [14]. It transfers only to a service with about 200 calls per second and 0.3 s p95 upstream latency [8]. At 200 calls per second, 80 threads saturate once API latency reaches 0.4 s [20]. The sizing leaves 20 threads above the 60 calls in flight [18][19]. Past that point we'd expect API stages to queue on their own executor, with database stages no longer waiting behind them [6].
The text of the post available to us ends before it reports a p99 after the change.
What to watch
- A post-change p99 and Hikari waiting-thread count from the team, which would show whether the split alone held.
- Whether the transaction that calls an external API is restructured so the connection is not held during the call.
- Any change to Hikari's maximum-pool-size, which would force the 18-thread database executor to be resized.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+10
- Incentives
- Not scored
- Confidence62
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Under load, the service's p99 latency climbed from 400 ms to 18 s while CPU sat around 20%.
- [2]
Every stage of every unit of work ran on one shared thread pool of 100 threads via CompletableFuture, and database access went through a Hikari connection pool with 20 connections.
- [3]
Thread dumps taken during the slowdown, lined up with Hikari metrics, showed nearly every pool thread blocked, with all 20 connections in use and up to 60 threads waiting for one.
- [4]
Blocked threads use almost no CPU, which explains the low CPU during the slowdown.
- [5]
One code path opened a transaction, called an external API and wrote the result before committing, so the connection stayed checked out for the whole API call; every slow response took a connection out of the pool while no query was running.
- [6]
The team split the shared pool into two executors, one for database work and one for external API calls, so each stage runs on the executor for the resource it blocks on.
- [7]
The database executor got 18 threads, set in code as Hikari maximum-pool-size (20) minus 2 for request-path queries that use the same Hikari pool outside the executor.
- [8]
The API executor got 80 threads, annotated in code as ~200 calls/s x 0.3 s p95 latency = 60, plus headroom.
- [9]
A database thread without a connection can only wait, so threads beyond the connection count add waiting and nothing else.
- [10]
With the sizes aligned, extra database work waits in the executor queue, where it holds no thread and no connection.
- [11]
Under load, slow external calls held their threads longer, leaving fewer threads for every other stage, including database work, which sat in the executor's queue.
- [12]
A thread waiting for a Hikari connection is just as unavailable as one waiting on HTTP, and API stages then queued behind such threads.
- [13]
A thread that needs the database first asks Hikari for a connection; if none is free, it waits.
- [14]
The team sized the API executor with Little's Law.
- [15]
The 18 s p99 is 45 times the 400 ms baseline.
- [16]
When 60 of the 100 pool threads were waiting in Hikari, 40 threads were not waiting on Hikari.
- [17]
The two executors total 98 threads, two fewer than the original 100-thread pool.
- [18]
About 200 calls per second at 0.3 s latency is 60 calls in flight.
- [19]
The 80-thread API executor has 20 threads of headroom over the 60 calls in flight.
- [20]
At 200 calls per second, 80 threads are all in use once API latency reaches 0.4 s.
- [21]
The post's summary says virtual threads would not have helped.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toThreads Are Not Connections: A Spring Boot Concurrency Incident
1 article · October 16, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.