Build1 distinct publisher3 min readPublished
A dev.to field notebook records forty-eight hours spent truncating logs on a filesystem that had gigabytes free and no inodes left, when two commands run in the right order would have named the cause on the first pass.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
`df -h` and `df -i` read two different counters on the same mount, and only one of them was ever going to answer this failure. The notebook's own pairing line puts them next to each other: `paste <(df -h / | tail -1) <(df -i / | tail -1)` [23]. That is the fix, and it is one command longer than the investigation that already happened.
The successful probe file is the part worth studying. `touch ~/probe-ok.txt` returned clean, and the author took that as evidence the worker path was somehow special [5]. What it established is narrower: one write, in one directory, at one moment, found a free inode. For the stronger reading in the post, that a filesystem can accept a file in one directory and refuse a two-kilobyte file in another while blocks remain [6], either the two paths sit on different mounts or the last inode went between the two attempts. The notebook lists `mount | awk '{print $1, $3}'` among the commands run and never reports what it printed [17]. The mount question is exactly what a pre-blame sequence has to settle before a successful probe means anything.
The remediation failed in a mechanical way. Truncating with `: > worker.log` [7] keeps the directory entry and the inode and returns only blocks, so it recovers none of the resource that was exhausted [21]. The safe-delete helper went further: as reconstructed in the post it copies every JSON sidecar into `runs/.trash/` with no unlink [9], taking N files to 2N inodes while the byte cost stays as small as the files themselves [22]. Copying before deleting is a polite way to reach the ceiling twice as fast. The worker duly started failing sooner, sometimes on file four [10][11].
The lens was the problem before the cleanup was. Cache trees, `.pytest_cache` and editor swap files never surfaced in size-ranked output because thousands of small nodes do not look large [8]. The author did count files in `runs/` once, found the number bigger than expected but not millions, and went back to watching bytes, which he names as the first mistake he would not repeat [19]. His own summary is that bytes and inodes answer different questions and he had asked one of them [14], and that `du -sh` would not have shown this in a form he trusted at 2 a.m. [15].
Hour twenty of forty-eight is about 42 percent of the incident spent before the command that named the cause [20]. The author's verdict is that `df -i` should have been second, not twentieth [16]. That is the portion I would take: not a monitoring project, just a fixed second look on any ENOSPC where `df -h` reports calm.
What has to be true for this to transfer is modest, which is why an n of one is enough to change an ordering. The workload has to write many small files, and the mount has to be able to run out of inodes before it runs out of blocks. The author claims no benchmark and no quota promise, and labels his reproduction as a lab rather than an outage [18]. That is the right way to publish a field notebook. The evidence here supports the ordering rule; it says nothing about how full anyone else's inode table is.
Ranked by verification strength, evidence, and original report placement.
The author watched a two-kilobyte write fail with "No space left on device" while df -h still showed free gigabytes, and then spent forty-eight hours cleaning the wrong evidence.
The traceback read: OSError: [Errno 28] No space left on device: 'runs/2026-09-05T07-12-04.json'.
The workload was a small Python worker that dumped JSON sidecars next to each run.
df -h reported plenty of space on the root filesystem, and /tmp looked equally relaxed.
The author created a dummy file by hand with touch ~/probe-ok.txt, it succeeded, and he told himself the worker path was special; he calls that touch the trap.
The author asserts that a filesystem can accept a file in one directory and refuse a tiny file in another while still having blocks to spare, and that inode exhaustion is the reason; he did not ask that question for twelve hours.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · September 4, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Shared memory in Java: the mmap is the easy half, the descriptor handoff is the work1 distinct publisher
build
Force the tool call, then hand Lightsail a long-lived key1 distinct publisher
build
A receipt intent written before dispatch turns an SMS timeout into a poll instead of a resend1 distinct publisher
build
A cache hit is a quota refund: semantic caching with trigrams and no vector database1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Checkable mechanism, single witness
Every fact about the incident comes from one dev.to notebook written after the fact, and it publishes no df -i output, no inode totals and no filesystem type, so the severity is stated as the author's claim, without supporting figures. What holds the account up is that its load of Linux behaviour does not depend on the author: a truncated log keeps its inode, a copy into runs/.trash/ adds one, and the included script lets any reader watch df -i move on a directory they own.
One mount, no signal
This is a single worker on a single mount. Nobody has reported the same failure alongside it, no team or deployment is described, and the author is explicit that nothing in the post is a benchmark or a quota promise, so there is no usage to measure.
Ending tidier than the trail
The framing hands the ending to a single run of df -i, while the notebook shows that reading only opened an ordinary hunt through per-directory and per-extension file counts before caches and sidecars surfaced. Pulling the other way, the author disclaims benchmarks, marks the inode-filling script as a demonstration, and answers 'not in a way I would have trusted at 2 a.m.' rather than claiming du -sh is useless.
Credit, not commerce
No product, vendor or pricing appears anywhere in the post. What the author stands to gain is credit for the confession, and that shows up in the headline and the second-person openings, while the technical content stays plain and unadorned; ': > worker.log' frees no inodes whether or not the piece finds an audience.
Reproducible in part
High for the parts a reader can rerun, low for the parts only the author witnessed. Nobody needs to corroborate that truncation releases blocks and keeps the inode; nobody at all corroborates the forty-eight hours, the hour-twenty df -i, or whether the failing path even sat on the mount he kept measuring.