Build1 publisher3 min readPublished
Watching a backup client from outside its own logs catches the variant that saved nothing
Before judging agent-written code, a dev.to author seeded a small C backup client with eight known behaviors and observed it with system-call traces, a loopback capture and a digest check to see which observer caught what.
The Engineer · Build desk

What happened
- A dev.to author built a small native backup client with AI assistance, gave it eight deliberately chosen behaviors, and observed it from outside its own logging.
- The stated contract is four clauses: save matching content, send no payload on an unchanged second run, contact only the configured receiver, and recover from an injected interruption within three attempts.
- Some variants produced the right file while doing questionable things on the way there, and one reported success without backing anything up.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint When an implementation and its own tests share a misunderstanding, their agreement settles nothing, so a review that wants independent evidence has to bring an observer sitting outside both.
- decision Anyone copying this harness has to write the requirement down first, because a system-call trace, a capture and a digest check convict only against a stated contract.
- capability A second loopback endpoint makes "it connected somewhere else" reproducible inside a test run, with no traffic leaving the machine and no third-party service involved.
- constraint The seeded design bounds what the results can be cited for: detection of known behaviors, not a defect rate for any model and not a speedup for any workload.
The first framing was tidier than the one that survived. Two sources of truth, the CPU and the network, according to the dev.to post published under the handle copyleftdev [25][5]. Then the objections arrived. A CPU can faithfully execute the wrong algorithm, and a packet capture can faithfully record the wrong bytes reaching the wrong destination [6]. Neither observation knows what the user actually requested [7]. What replaced it is narrower: execution, network activity and resulting state are three bodies of evidence with different coverage and different blind spots, and they only mean something when judged against a requirement [8].
The requirement here is that the saved backup must match the source file [9]. That is decidable because the input is a deterministic 512 KiB file, the client computes a SHA-256 digest, and a separate verifier computes the saved file's digest and compares it with the source [10][11][12]. The verifier never reads the client's logging [2].
Each scenario runs the client twice against the same receiver state, and the baseline uploads on the first invocation and recognizes unchanged content on the second [13]. The contract requires no upload payload on that second invocation [14]. 512 KiB is 524,288 bytes [24], so a broken content check shows up as half a megabyte of payload on a loopback capture. You can count the bytes without decoding them.
One rule sits on the receiving end. An upload whose payload does not match its advertised digest must be rejected before it replaces the saved backup [15]. A client that advertises one digest and sends different bytes is caught by the receiver, which validates incoming content before committing it [12]. The "contact only the configured receiver" clause is checked with a second local endpoint, so an unexpected connection can be demonstrated without contacting an outside service [17].
Repeated hashing and spinning are in as diagnostic cases for unnecessary work. The post does not set a performance threshold or measure speedup [16]. The question behind the exercise, quoted from Hemapriya Kanagala's article on evaluation, was "92% according to what?" [4].
The eight behaviors are deliberately seeded demonstrations whose job is to check whether the observers detect known behaviors [18]. The post is explicit that they do not measure how often an AI agent introduces these defects and are not evidence about a particular model's coding ability [18]. For the file comparison to transfer to a real client, the source data has to hold still while it is being read, which a deterministic 512 KiB fixture file does by construction [10].
The author wrote that "familiar frameworks do not automatically validate unfamiliar code assembled on top of them" [22], and that "if the implementation and its tests share a misunderstanding, agreement between them can be misleading" [20]. The stated worry is the volume of plausible implementations now available against the time to examine them [23]. Hence the standing request: the author says that a claim something works increasingly gets answered with a request to see the packet capture [21].
What to watch
- Whether the remaining evidence collectors beyond strace get published, and which seeded behavior each one missed.
- Whether the same fixture is run against unseeded agent-written implementations. That is the only version of this test that says anything about models.
- Whether a performance threshold is added, since no speedup has been measured for the repeated-hashing and spinning cases.