Skip to content

Build1 publisher3 min readPublished

Five stacked defects pushed a SaaS dashboard's p95 from 400ms to 4.2 seconds

After bigger instances and a read replica failed to hold, a scheduling SaaS spent its first week tracing requests and changed no code. The five defects it found were each small enough to have shipped unnoticed.

The Engineer · Build desk

Photograph accompanying Five stacked defects pushed a SaaS dashboard's p95 from 400ms to 4.2 seconds
Photo: runn.io

What happened

  • A scheduling and resource-planning SaaS with 40,000 users watched p95 response time climb from 400ms to 4.2 seconds over roughly six months.
  • Bigger instances, a read replica and scheduled restarts had all been tried before the investigation started, and none of them held.
  • The first week went entirely to instrumentation, tracing requests across the API gateway, app servers, database and cache layer under normal load with no fixes shipped.
  • Five separate defects surfaced, and the writeup says none of them alone accounted for 4.2 seconds of latency.
  • The reported end state is 380ms, which is 3.82 seconds faster than the worst measurement and 20ms below where the system had started.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The named savings cover at most 59 percent of the total improvement and are quoted in three different units, so anyone copying this playbook has to re-measure in their own percentiles before promising the top-line number.
  • decision A team without two failed scaling rounds on record will be told to try capacity first; the evidence for skipping it here is a negative result somebody already paid for twice.
  • capability Once the analytics call moved behind a queue, the provider went down twice during the engagement without users seeing it, so third-party availability left the user-facing path.
  • cost The method costs a week of measurement with nothing to show, plus one release per fix, which is the price of knowing which change bought which milliseconds.

Add up the savings the writeup itemises and you do not reach 3.82 seconds. Moving the analytics call to a queue took 800ms to 1.5s off the median request on the affected endpoints [14]. The dashboard query fell from about 620ms to about 45ms, worth roughly 575ms [16][2]. Connection wait at peak fell from about 180ms to under 5ms, worth about 175ms [19][3]. That is between 1.55 and 2.25 seconds of named savings against a p95 improvement of 3.82 seconds [1][4]. The figures also sit in three different units: the top line is p95, the analytics saving is quoted against the median, and the query and connection numbers are averages [1][14][16][19].

The rest of the tail most plausibly came from the cache. A 34 percent hit ratio means 66 percent of those lookups reached Postgres; at 91 percent it is 9 percent, so the path now sends about 14 percent as many reads to the database, an 86 percent reduction [20][5]. On that change the author wrote, "Small diff, outsized effect on database load." [21]

One number does not survive a check. The writeup says about 40 percent of Redis calls crossed availability zones at 2 to 4ms each, and that at volume this cost hundreds of milliseconds per request [11]. For 40 percent of calls to add 200ms at the top of that penalty range, a single request has to make about 125 Redis calls; at 3ms it needs 167 [6]. The writeup gives no per-request call count [26].

The shape of the list explains why the earlier moves did nothing. Exactly one of the five findings is a read-capacity problem, the cache misses landing on Postgres [20][8]. The other four are round-trip count [8], a pool statically sized above the database's own limit [9], network placement [11], and a blocking outbound HTTP call [12]. A read replica does not shorten 41 round trips, and a larger instance does not make an external analytics provider answer faster [8]. The team treated two rounds of vertical scaling with zero improvement as the finding and left instance sizes alone after that [24].

The pool arithmetic is the plainest defect. Eight app servers at 20 connections each is 160 possible against a max_connections of 100, which is 60 more than the server will accept [9][7]. A pool configured that way fails the same way on a bigger instance. The fix cut per-server pools to 12, giving 96 [17], but the line worth reading in the PgBouncer block is default_pool_size = 25 with reserve_pool_size = 5, so the server-side ceiling is now about 30 connections while max_client_conn = 500 absorbs the app tier [18][7].

For the top-line number to transfer, a system needs the same four shapes: a synchronous third-party call inside the request cycle [12], a page that issues one query per child record [8], static per-server pools above the database limit [9], and namespace-wide cache flushes triggered by unrelated writes [10]. Remove the analytics call and the item the writeup's own priority table ranks first, high impact and low risk, is not available to you [23]. The author's account of the cause is accumulated technical debt in a system that grew for a few years without a dedicated performance pass [27].

What to watch

  • Whether the 91 percent cache hit ratio holds past the first week now that the 30-second TTL is a safety net and not the primary invalidation mechanism.
  • Whether default_pool_size = 25 survives the next app servers added, since the ceiling now lives in PgBouncer instead of the per-server pool.
  • Whether anyone reproduces the cross-zone Redis figure with a documented count of Redis calls per request.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories