Replay files must record prompts, tool calls and decisions before an agent patch is merged

A stored sha256 of stdout lets you check a later run against the first one, without handing you the bytes. The post's storage rule is explicit about that trade: store stdout hashes when the bytes are large, and raw text only when the payload is tiny and non-secret ^[5]. Its definition of replay is stronger, since it asks that stored inputs reconstruct the same side effects ^[2]. A digest reconstructs nothing. What it gives you is a comparison, so you learn that a second run read different bytes at call seq 4 than the first one did ^[6].

The model_id field is a pinned opaque string, and the post tells reviewers to watch for model labels copied from a pricing page ^[7]. Pinning the label still leaves the sampling loose. None of the example's ten top-level keys records a temperature, a top_p or a seed ^[6]. Two runs under one model_id can emit different diffs and both satisfy the checker's required set ^[8].

The post writes that "If any row is missing, you do not have a replay. You have a diary entry with extra punctuation." ^[4] Its own checker requires nine keys ^[8], and the example JSON carries ten ^[6]. The one it omits is task_id, so a replay file with no link to an issue passes ^[9]. The published excerpt also stops mid-condition, at `if len(argv) !` ^[11], so the reader never sees what the script does with patch_digest. The docstring says "Example only: unexecuted replay-file checker. Not a benchmark." ^[10] The post adds, "Do not treat a passing checker as proof of model quality." ^[12]

The vocabulary is unsettled too. The six rows name the human decision as merge, reject, or isolate ^[13]. The example sets human_decision to "review" ^[14]. That value sits outside the three ^[15].

The allowlist beside the JSON file is what keeps the instrumentation tractable: read_file, write_file, run_tests, with unknown names failing the run ^[16]. Three named functions are a small surface to wrap with digest capture. A shell tool is an unbounded argument space, and the record then holds a hash of a command string whose effects were never enumerated. The post keeps broad shell out of the first draft ^[16].

The post reports no session counts, no artifact sizes and no CI timings ^[21], and it calls the schema an example, not a shipped standard ^[17]. The disclosure at the foot says the article was prepared as part of MonkeyCode's product outreach, and that MonkeyCode offers free model access and a free server option ^[18]. On that free tier it holds the bar it set, telling readers to keep the same checker when they later pay for inference ^[19]. Whether the artifact reaches the reconstruction its own definition promises depends on how much of your tool traffic is small enough to store raw and non-secret ^[5].