Build1 publisher3 min readPublished
No source edits until a flake repeats twice, tested across a 16-case environment matrix
A dev.to field report spends 48 hours on one flaky test and ends at environment drift. What it hands over is four written constraints and a grid of locale, timezone, worker and file-descriptor settings.
The Engineer · Build desk
What happened
- The author set four constraints before debugging, starting with no source edits until the same failure appeared twice under a recorded command.
- By hour six the failure was tracking conditions rather than code, since a single-threaded local run under the author's default locale never failed at all.
- The harness at tools/repro_matrix.py grids LC_ALL, TZ, worker count and the nofile limit across two values each, writing one JSON line per case to artifacts/matrix.jsonl.
- Moving the grid to a fresh machine narrowed the failing conditions and pointed at environment drift instead of the application logic.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Satisfying the two-failure rule costs up to eight hours of wall clock at the published 900-second timeout, so a team adopting the protocol is buying machine time.
- capability Because each case appends a row carrying exit code, timing, stderr tail and overlay, a colleague can check a reproduction claim without the person who ran it being in the room.
- decision Anyone copying the sequence has to decide what supplies the clean witness, since a container built from the lockfile does the same job as the vendor's free server.
- constraint The write-up can only be adopted as a procedure; there is no diagnosis in it to copy into your own repository.
Call run_case and it merges the case overlay onto os.environ, starts a monotonic clock, and hands the command to subprocess.run with capture_output, text mode, and a 900-second timeout [13]. What comes back is one row per case: exit code, elapsed time, the tail of stderr, and the overlay that produced them [2]. The log opens in append mode, so a second sweep lands under the first in artifacts/matrix.jsonl instead of overwriting it [14].
Four dimensions with two values each is 16 cases [15]. At the 900-second ceiling, one sweep can run four hours [16]. The first constraint wants the failure twice, so plan for eight [17]. That fits the 48 hours the author allotted [6], and it is the figure to schedule before promising anyone a reproduction.
Capture came first. The failing command was copied verbatim out of the CI log, run locally, and the environment snapshotted next to the result instead of reconstructed from memory later [7]. The snapshot is one timestamped file holding uname -a, python -V, pip freeze, locale, and ulimit -a [8].
The reframe the post recommends is to stop asking why it fails and ask "fail relative to what?" [10]. Each hypothesis was written down before it was tested, because untested theories that live only in someone's head survive too long [3]. The captured artifacts went to a model: trimmed stderr, locale output, dependency list, exact command, with a request for ranked causes and the one observation that would falsify each [18]. Roughly a dozen came back and most died cheaply [22]. A kill test gated every trip into source, and only artifacts went into the prompt [21].
"My laptop is a terrible witness," the author wrote [23], listing stale virtualenvs, a warm kernel page cache, and a .env file forgotten since 2024 [24]. None of that is in the repo. The clean run cloned the project, installed from the lockfile, and ran the harness with nothing else on the box [25].
The article was prepared as part of MonkeyCode's product outreach [19], and the two steps it credits to that vendor are the model access and the clean server [18][25]. The author writes that he did not independently verify quotas, limits, or how long any of it stays free [20]. In my view those are also the two substitutable steps: a container built from the lockfile is a clean witness too, and the constraints do not care who ranks the hypotheses. The artifact format is the part with no substitute, because it is what a second engineer reads when the test passes on their machine [2].
The published text does not name the test or state a root cause. It stops at a narrower set of failing conditions on the clean box and points at environment drift [26]. The harness is offered as "a template, not a benchmark" [27], with the matrix swapped for whatever dimensions your own failure seems to track [11]. The author calls the fourth rule the one people skip: at hour 48, an unexplained failure gets a written summary and a follow-up ticket instead of a third night [4][5].
What to watch
- Whether MonkeyCode's free token allowance and free server option still exist, and at what quotas.