Build1 publisher3 min readPublished
A sleeping laptop expired a 30-second HTTP budget by half an hour
A queue worker built its remaining-seconds budget from time.time(), and after the host woke from sleep it logged remaining=-1842.7. Two days of debugging went to sockets, Docker and NTP before the clocks got compared.
The Engineer · Build desk
What happened
- A queue worker capped stuck HTTP calls with a remaining-seconds budget built as deadline = time.time() + 30, so every later check subtracted the current wall clock from a stored calendar point.
- After the laptop woke from sleep, a job that should have run for twenty seconds returned immediately, and a second job sat inside an HTTP call far longer than the budget the author thought was configured.
- The log printed remaining=-1842.7 once, and the author treated it as a formatting problem, adding more decimals to the line.
- Debugging went through HTTP client prints, a library swap, a retry wrapper that still used wall time, packet traces, container CPU limits and a grep for sleep( before the clock was examined.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Two days went to client prints, library swaps and packet captures, and none of them could move the number in the log.
- constraint A test suite that never suspends the host or moves the clock cannot produce a negative remaining value, so the failure leaves CI green and surfaces only on a machine that sleeps.
- decision Every existing deadline site now needs a call on which clock it reads, and on whether the budget should keep counting or pause while the host is suspended.
- exposure Teams accepting a timeout helper from a model's first draft inherit the wall-clock pattern, and on a server that never sleeps it stays latent until the clock steps for some other reason.
`deadline = time.time() + 30` stores a point on the calendar, not a duration [2]. Every check after that subtracts whatever the calendar currently says from that stored point. Suspend the host and the calendar resumes at the real current time, hours ahead, so the subtraction returns a large negative number and every budget in flight is already expired [14]. `time.time()` can also step backward [11]. When it does, the same subtraction returns a bigger remaining value and the budget stretches instead of firing [22].
The single logged `remaining=-1842.7` [5] puts the check 1842.7 seconds past a deadline that was set 30 seconds out, so 1872.7 seconds had passed since the budget started, about 62 times the budget it was meant to enforce [20]. That is roughly half an hour [21], while the post describes the clock jumping forward by hours [14]. The author filed it under logging and padded the format with extra decimals [6].
The first command in the notebook printed all three clocks side by side [10]. `time.time()` is seconds since the Unix epoch and can jump in either direction [11]. `time.monotonic()` starts from an arbitrary baseline and only moves forward, which is what a duration budget wants [12]. `time.perf_counter()` is monotonic as well and better for short intervals, though the post says a thirty-second HTTP budget does not need that resolution [13]. That print runs once. Run it before the lid closes and again after the wake and you would see whether the monotonic clock advanced while the host was down.
Before the clock, the suspects were the network, then NTP, then Docker, then the helper [23], and the commands include a `pmset` wake/sleep grep and a `systemd-timesyncd` log dump [24]. Time sync is the wrong layer for this failure. A perfectly synchronised wall clock still steps when a host suspends and resumes [14], and the budget is anchored to a value that step moves.
The unit tests passed for a structural reason: they never slept, never warped the clock, and never crossed midnight [7]. A suite that only ever reads a forward-moving clock cannot produce a negative remaining value. The author wrote: "I did the usual noisy debugging during this stretch, and almost none of it actually touched the clock" [9].
The post discloses that it was prepared as part of MonkeyCode's product outreach, and that the always-on box used for the overnight reproduction was MonkeyCode's free server option [15]. The author also used the free model access to draft extra failing tests, then edited them by hand [16]. The first helper the model drafted still called `time.time()`, which the author attributes to that pattern's presence in nearly every snippet dump, and says it was almost merged onto a machine that never sleeps [17].
What to watch
- Whether the author publishes the kept helper's final code, since the post breaks off mid-sentence before it appears.
- Whether the repro gets run as a before-and-after print across an actual host suspend. That print shows what the monotonic clock did while the machine was down.
- Whether the failing tests drafted on the free model access end up in CI with a clock the test can move.