Build1 distinct publisher3 min readPublished
nginx resolves a literal proxy_pass hostname while it parses the config, and that parse runs inside ExecStartPre, so a one-second DNS gap during an unattended upgrade was enough to keep the unit from starting.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Here is what actually happens when nginx reads `proxy_pass https://example-backend.tld/api/lead-capture`: it resolves that name during config parsing and treats a lookup failure as a fatal configuration error [7]. The parse that matters here is the one the packaged unit runs as `ExecStartPre=/usr/sbin/nginx -t`, so a name that does not resolve for one second means the process never reaches the point of binding port 80 [8]. The request path was never involved at all; the site had no traffic to that backend at 06:49, only a config test [5].
The window came from the upgrade itself. `apt-daily-upgrade.service` started at 06:49:44, systemd requested re-execution nine seconds later because it had just upgraded itself, and `certbot.timer` fired in the same second [10]. According to the author, the re-exec took the local resolver with it momentarily, and anything restarted inside that gap had to survive without DNS [11]. This was the default upgrade timer on a default Ubuntu install against a default nginx package; the only odd thing was the hour [14]. Which is a poor hour for a nightly job, mostly on the grounds that it is not night.
The arithmetic is where the story stops being about DNS. The lookup failed for about a second and the outage ran seven hours and forty minutes [19], roughly 27,600 seconds of downtime for one second of unavailable name service [4]. A watchdog on another site alerted 9 minutes 26 seconds after nginx stopped [2]. The human start came 7 hours 30 minutes 12 seconds after that alert [3], which is about 98 percent of the total [6]. For the seven hours to transfer to your box you need two properties, not one: a literal upstream inside an enabled site file, and nobody acting on the page between 07:00 and 14:30.
The config fix is documented and cheap. Put the upstream in a variable and nginx defers resolution to request time, which requires a `resolver` directive; the author uses `resolver 127.0.0.53 valid=30s ipv6=off` with `set $backend` [15]. The trade is the point: a startup DNS failure takes every page on the server, a request-time DNS failure takes one endpoint [16]. An upstream written as an IP or a unix socket carries no DNS lookup at startup at all, so it never becomes a startup dependency [18]. Auditing for this is a `grep -rn "proxy_pass"` across both `sites-available` and `sites-enabled`, and the author notes his first grep returned nothing because it had not followed the symlinks [17].
Above that sits the packaging default. nginx ships with `Restart=no`, so a failed start stands until a person intervenes [20]. The drop-in sets `Restart=on-failure`, `RestartSec=15`, `StartLimitBurst=40` and `StartLimitIntervalSec=600` [21]. Note the numbers: 40 attempts at 15 seconds is exactly 600 seconds [5], so the retry budget and the rate-limit window coincide at the boundary, which is a thing to reason about before an incident rather than during one. The author's stated reason for capping it is that an unbounded restart on a genuinely broken config is a service lying about its state [23]. He also says the policy has so far been verified only by reading `systemctl show` [24]. Until someone breaks resolution deliberately and watches the unit come back, that drop-in is a claim about documented behaviour, not a tested recovery path.
Ranked by verification strength, evidence, and original report placement.
A scheduled syndication job that publishes one blog post a day failed four times at 10:00 with "Server error '521 <none>'" for the URL https://neuragrowth.co/blog/schema-grammar-ceiling/.
HTTP 521 is Cloudflare reporting that the origin server did not answer.
By the time the 10:00 failures were noticed, the author's whole site had been down for over three hours.
The host itself was healthy: four days of uptime, load under 0.2, disk at eight percent, but systemctl is-active nginx reported failed and nothing was listening on port 80 or 443.
The journal recorded at 06:49:54 that nginx.service was stopping, that nginx emitted "[emerg] host not found in upstream \"example-backend.tld\" in /etc/nginx/sites-enabled/site:104", and that the configuration file test failed.
Line 104 was a location = /api/lead-capture block with proxy_pass https://example-backend.tld/api/lead-capture, added months earlier so the public site could forward one form endpoint to a backend on another host without revealing its name.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · September 1, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Build concurrency on one VPS is a division problem, and the app you serve pays the remainder1 distinct publisher
build
systemctl restart reopens the app port 2.9 seconds after the atomic symlink swap1 distinct publisher
build
SSE in Go breaks twice before your handler runs: an illegal header, then a 30-second timeout1 distinct publisher
build
796 pages of semantic search with no vector database, and what it cost to skip one1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Quotable logs, one witness
Three journal lines and two config blocks carry this account, and they are specific enough to argue with: host not found in upstream, configuration file test failed, unit stopped at 06:49:54. What a reader cannot check is the hinge — nothing quoted shows a lookup failing or the stub resolver going away, that is inferred from everything happening in the same second. The underlying behaviour is the sturdy part: parse-time resolution of a literal proxy_pass target inside ExecStartPre, on a package that ships Restart=no, reproduces on any spare Ubuntu box. The incident wrapped around it has one witness and a redacted hostname.
One box, self-reported
Counted honestly, the footprint is a single machine: the variable-and-resolver rewrite is shown installed, the alerting change is described as done, and the restart drop-in exists but has never been exercised. No fleet rollout, no second operator reporting back that the proxy_pass grep found something, no vendor picking up the pattern. The advice may generalise widely; the observed practice does not yet.
Talked down, not up
The one stretch runs in the modest direction and against the author: the headline says eight hours offline while the timeline adds to seven-forty. Everywhere else the piece deflates itself — the restart policy 'goes on the list rather than in the win column' precisely because killing nginx to test it would cost real downtime. Meanwhile the transferable finding, that any literal hostname in proxy_pass is a boot-time dependency on DNS for the entire server, gets less drum-roll than it deserves.
Nothing to sell
One developer writing up their own downed site on dev.to. No product appears, no vendor is flattered, and the only commercial artefact in the story — the author's own domain — shows up inside an error message. The pull that does exist is reputational: incident postmortems build a byline, and a tidy three-lines-of-journal narrative is the shape that travels. Set against that, admitting the safety net is a belief rather than a net is the least flattering sentence available.
Lesson firm, story unverifiable
We would repeat the engineering advice without hedging: the parse-time resolution, the ExecStartPre parse and the Restart=no default hold on any stock install and are easy to confirm. We would hedge the narrative. Whether the outage really ran 06:49:54 to 14:29:32, whether a systemd self-upgrade briefly removed the resolver, and whether the neighbouring watchdog was the only thing that noticed all sit inside one person's reading of their own logs, with no second observer and an anonymised backend.