Build1 distinct publisher3 min readPublished
The reload endpoint returned 200 and the logs agreed, because Prometheus really did reread the file it had open. The host's file had been replaced by a new inode, and a single-file bind mount never follows the name.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The four steps inside `sed -i` are the whole failure. It opens the target for reading, opens a temp file in the same directory, streams the transformed bytes into it, and calls `rename(2)` to move that temp file over the target name [7]. All four succeed. The path on the host now resolves to a fresh inode carrying the new content, which is exactly why `md5sum` on the host looked correct [4]. The mount was established when the container started, against the inode the name pointed at then, and single-file bind mounts are bound to inodes rather than to directory entries [8]. The rename moved the name, but the mount stayed with the inode.
Both `stat` outputs report the same device, fc01h/64513d [5][6]. One filesystem here holds two live inodes, and no copying is involved [3]. The container's inode reports `Links: 0`, which means no directory entry anywhere on that filesystem points at it, and the only thing keeping it from being freed is the descriptor the container still holds [9]. That field means no directory entry points at this file anymore except through the descriptor your process holds.
The sizes deserve a minute of arithmetic. The host file is 2847 bytes and the file the container reads is 2791 [5][6], a gap of 56 bytes [1]. The substitution shown in the post swaps `15s` for `30s`, three bytes for three bytes, so it moves the size by zero [2]. Whatever produced 56 bytes of divergence is therefore not in the command list the post publishes, which points at an earlier atomic write having already detached the mount well before anyone started debugging this reload.
The escape the author used first is `cat > file`, which opens the existing inode with `O_TRUNC` and writes into it, leaving the inode number unchanged so the container sees the new bytes at once [10]. That transfers only if two things hold: the writer can open the existing inode, and you control every tool that ever writes that path, because anything ending in a rename puts you back where you started [11]. It also gives up the property `rename(2)` was invented to provide. `O_TRUNC` empties the file before the new content lands, so a SIGHUP arriving in that window reads a short config [4]. The second escape, `docker compose up -d --force-recreate prometheus`, keeps atomic writes and pays a container restart instead, and the author reaches for it when the config change is paired with an image bump, an env-var change, or a sibling edit [12].
In my context I take the recreate for anything a process rereads on signal. A few seconds of scrape gap is bounded and visible in the graphs. A ghost inode is neither, and it cost this team an hour before they thought to run `stat` [13]. The cheap instrument here is comparing inode numbers across the container boundary rather than checking content on the host [11]; the expensive habit is trusting that a 200 from a reload endpoint says anything about which bytes were read.
Ranked by verification strength, evidence, and original report placement.
The Prometheus container ran the prom/prometheus image with its config bind-mounted as /opt/impress/prometheus.yml:/etc/prometheus/prometheus.yml.
The team ran sed -i 's/scrape_interval: 15s/scrape_interval: 30s/' on the host file, then ran docker exec prometheus kill -HUP 1 to signal a config reload.
Prometheus's /-/reload endpoint returned 200 and its logs said it had reloaded the config, but the runtime config in the UI still showed scrape_interval 15s.
md5sum on the host returned a3f9c1e8... while md5sum inside the container on the mounted path returned 7b2d4f06..., so the container was reading different bytes.
stat on the host file reported Size 2847, Blocks 8, IO Block 4096, Device fc01h/64513d, Inode 1835421, Links 1.
stat inside the container on /etc/prometheus/prometheus.yml reported Size 2791, Blocks 8, IO Block 4096, Device fc01h/64513d, Inode 1835098, Links 0.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · September 1, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
The 300-monitor wall is real, and sharding Uptime Kuma is a bill you keep paying1 distinct publisher
build
Eight containers against a variable APM bill: what the self-hosted stack actually costs you1 distinct publisher
build
Eleven agent sessions on one machine settled CPU contention by writing to each other1 distinct publisher
build
912MB to 108MB is mostly typing now, and that weakens the base-image excuse in review1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Textbook mechanism, one unverified paste
The kernel behaviour at the centre of this is checkable by anyone with two terminals: rename(2) leaves the mounted inode orphaned, Links: 0 survives on a single open descriptor, and dev.to describes that chain correctly. The incident wrapped around it is another matter — one author, one set of md5sum and stat output, nobody who reran it, and byte counts that the command shown above them cannot produce.
One container, one team
The entire deployment record here is the author's own Prometheus service, fixed with --force-recreate and reported after the fact. The long list of atomic-rename writers the post assembles is a list of tools, not of anyone who got bitten, and nothing indicates how common the failure is across teams running single-file mounts.
Right diagnosis, softly sold fix
The headline holds up — atomic writes and single-file bind mounts really are incompatible, and the reload endpoint really was telling the truth. The overreach is quieter than the title. Truncate-and-write is presented as one of two 'correct ways out' with no hint that emptying the file first hands a scraper a chance to read a stub, and the reproduction as printed cannot yield the file sizes it prints.
War story, nothing on sale
A developer's incident writeup cross-posted from a personal blog to dev.to — no product to move, no vendor to flatter, no benchmark to win. The pull that remains is the format's own: a clean, memorable failure makes the better post, and a clean narrative is exactly what gets in the way of admitting that the byte counts don't line up.
Act on the lesson, discount the numbers
Two forces pull opposite ways. The mechanism is bedrock and independently reproducible, so the advice is safe to adopt tomorrow. The specific incident is a single self-report with an internal inconsistency nobody has resolved, so the pasted digests, sizes and inode numbers deserve to be read as illustration rather than as measurement.