Build1 publisher3 min readPublished
Datadog billed 5.6 times what the two servers replacing the AWS cluster cost
An audit of a post-Series A fintech serving 35 requests a second puts one observability line at $3,800 a month, more than five times the two bare-metal machines that replaced the whole AWS cluster.
The Engineer · Build desk
What happened
- A consultant auditing a post-Series A fintech found a monthly AWS bill hovering around $14,200 against edge traffic that peaked at roughly 35 requests a second during business hours.
- The console held fourteen microservices on two dozen EKS pods, an Aurora multi-AZ cluster with read replicas the team never touched, three NAT gateways and managed MSK between colocated services.
- The post lists a Datadog integration ingest-logging every health-check ping at an additional $3,800 a month.
- After four weeks the workload moved to two bare-metal servers from a commodity provider at $340 a month each, one primary and one warm standby in a separate facility.
- The author reports the bill fell to around $720 a month and average API p99 latency from 145 milliseconds to 11 milliseconds.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The recurring difference is about $13,480 a month, and it is bought with in-house operations work: the two machines, their mirrored NVMe and the standby are now the team's to run.
- constraint Because the Datadog figure is described as additional, a reader cannot pin total spend at $14,200 or about $18,000, and cannot rank the NAT gateways against the idle Aurora replicas by cost.
- decision Anyone copying this design is choosing an approximately five-second recovery point for a payments ledger, and choosing to write the failover procedure themselves.
- capability For teams whose p99 is spent in inter-service hops and TLS setup, this audit puts a measured number on what collapsing those hops recovers.
Two servers at $340 a month each is $680 [1]. The post says the new bill covers offsite backup storage and DNS routing as well as the machines [12], so about $40 a month pays for everything that is not compute [2]. The Datadog integration on the old stack was $3,800 a month [5], about 5.6 times what the replacement hardware costs [3].
The post describes that $3,800 as "additional" [5], which leaves two readings. Count it inside the $14,200 and $10,400 a month sits behind EKS, Aurora, the three NAT gateways and MSK together [5]. Count it on top and monthly spend was about $18,000 [6]. Either way, deciding which object to kill first needs a per-service breakdown from your own bill.
The performance claim is the part that travels least well. The post attributes the 134 ms it took off p99 [7] to nine internal network serialization hops, two software load balancers and half a dozen TLS handshakes that ran every time a user fetched an account profile [14]. If the nine hops carried all of it, each hop was costing about 15 ms [8]. For the number to transfer, your slow endpoint has to be slow because of hop count. When the 145 ms is one unindexed query, collapsing fourteen services into one leaves the query where it was.
Kafka came out because the asynchronous workflow moved into Postgres with SELECT ... FOR UPDATE SKIP LOCKED, which the author says happened "without adding a single millisecond of latency" [9]. That was measured against traffic peaking near 35 requests a second, about 2,100 a minute [2][3]. The author's yardstick for that load is a Raspberry Pi 4 running a bare-bones Go HTTP server on a kitchen table [17]. Reusing the Postgres queue asks that your event volume fit one write path and that your consumers tolerate polling.
Durability is where the design gets genuinely opinionated. Backups stream continuous WAL segments to an offsite S3-compatible bucket through pgBackRest, which the post credits with a deterministic point-in-time recovery window of about five seconds [11]. For a fintech ledger, five seconds of committed writes is what a primary loss can cost. The second machine is a warm standby in a separate facility [7], and the post does not say how failover to it happens.
The teardown itself is careful work: fourteen services into a modular monolith [8], Datadog swapped for structured logging into a local VictoriaMetrics instance [10], four weeks start to finish [6]. The editorial case around it is broader than the evidence. The cloud, the author writes, "was invented to turn capital expenditures into predictable operational expenditures" [15]. The post also draws its own boundary: "If your system actually handles 200,000 concurrent writes per second and requires physical geographic redundancy across three continents, by all means, pay the cloud tax." [16]
What to watch
- Whether a per-service cost breakdown is published that prices the three NAT gateways, the idle Aurora read replicas and the MSK cluster separately.
- Whether the failover path to the warm standby is documented, and what recovery point it holds when the primary is lost.
- Whether the Postgres SKIP LOCKED queue is measured above the 35 requests a second it was tested at.