The model was fine: a 3-second P99 at 740K ops/sec was reads queueing behind writes

The arithmetic is what makes this concrete. At 740,000 operations per second, a P99 is not a handful of unlucky users: it is roughly 7,400 operations every second sitting in the slow bucket ^[13]. While one of those requests waits out its three seconds, about 2.2 million others pass through the same system ^[14]. In a trading app, that tail is the product.

The mechanism is unglamorous. Each inference call was doing only a handful of reads, and those reads queued behind writes once concurrency rose ^[4]. Cheap reads, expensive queue. Past a certain write throughput you get lock contention, and at that point retries, bigger caches and connection pool tuning stop moving the number ^[5]. Averages stay presentable because the median request never meets the lock; Koopmans is blunt about what that means, calling tail latency "a property of your architecture" rather than a bug, and pointing at storage-engine GC pauses landing at exactly the wrong moment ^[6].

The named culprit was Postgres under pressure, which he is careful to describe as a database asked to do too much rather than a slow one ^[7]. Read that as a placement error, not a product verdict. A read path with a hard deadline was sharing a store with a write path that had none, and the deadline lost.

The accuracy half of the story has the same shape. Wallet-address profile staleness blew past a five-minute SLA by hours while offline evaluation metrics stayed clean the whole time ^[8]. If "hours" means even two, the freshness target was missed by a factor of 24 ^[15]. Koopmans describes the result as metrics that look great offline and performance that is "rubbish" once online data is mixed in ^[9]. An offline harness scores a model against features it is handed; it never has to fetch them, so the one variable that broke production, the age of the feature at read time, is the variable the harness holds constant.

Then the index. Recall fell to 42 percent, meaning approximate search was missing 58 percent of true best matches, and query latency grew at the same time ^[16][10]. That combination is worse than either failure alone, because the symptom looks like a performance problem and is also a correctness problem. Worse, the trigger was improvement: re-embedding after each model change added mutations to the HNSW graph and rotted it further each pass ^[10]. Any change to the similarity function, the search parameters or the embedding model means starting the graph from scratch ^[11], so the accuracy work funds its own regression unless someone budgets the rebuild.

One caveat on weight. This is a single engineer's account of a single ML-based financial trading app, shared in conversation with a colleague at the same vendor ^[1]. The numbers are self-reported and the failure modes are common enough to be credible, but nothing here is a benchmark.