Only one of the six recorded fields is guaranteed to move between two correct re-runs of the same evaluation, and that is the elapsed time, reported as 120.0 milliseconds against an exit code of 0 in the post's own sample artifact [10]. Five fields are left worth comparing, which makes checking one run against another a field-by-field exercise rather than a diff of whole files [14].
The fields that are absent matter just as much. The digest the repository generates is taken over the evaluation configuration, binding the YAML to the artifact it produced [9]; nothing in the recorded fields identifies the image, the interpreter build, or the resolved package versions [13]. The artifact tells you the SBX executor was selected [8]. What was inside the sandbox at that moment is a file somebody else owns.
Set that against the post's own diagnosis, which names three drift sources: Python dependencies that change, local tools that go out of step, and setup steps that nobody wrote down [2]. The third one this design closes, because the YAML definition is the setup step, executed instead of described [6]. The first two stay wherever the sandbox image comes from.
That is still a real gain, and worth being precise about why. A page describing how to run an eval is not the same as documenting it. The post's own distinction is sharper than that: a written evaluation records what someone intended to do, and an execution-backed one records what actually happened [16]. Most reproduction attempts die in the space between those two, and writing down the command and the exit status closes it without anyone relitigating the judge model. Docker's complaint is that attention has gone to benchmarks and scoring methods while how a run gets executed receives much less [3].
Two axes sort the decision for whoever owns this on Monday: whether you can re-run the exact command and flags that produced last quarter's number, and whether you can rebuild the environment it ran in. If both are no, the runbook only works because you take it on faith. Command yes and environment no is where this kit puts you, which is sufficient if your failures have been about invocation, a wrong flag or a stale script path. Environment yes and command no is a pinned image with a lost invocation, and that is the cheapest case for the kit to fix. Both yes is the goal, and it requires an image pinned and versioned by a person you can name, with the artifact recording that the sandbox was used rather than what was in it [13].
One detail hints at where this goes. The repository is packaged so it can be applied when starting a Claude sandbox, with `sbx run claude --kit .` [11], which puts the eval harness and the agent's working environment behind the same description. The kit covers the execution half of reproducibility; the environment half still comes down to an image with an owner.
