Build1 publisher3 min readPublished
Rebuilding an agent sandbox from its declared recipe leaves the failed apt install behind
The author of a dev.to series on agent sandboxes reached for podman commit first and then rejected it, and now keeps a finished environment as a text recipe the privileged supervisor owns and rebuilds from a fresh base.
The Engineer · Build desk

What happened
- The author's first instinct for keeping a working agent sandbox was podman commit on the container to turn it into a reusable image, and the post calls that move a trap.
- Three reasons are given against the snapshot: it captures debris, it can freeze secrets that were on disk, and it becomes an opaque artifact the operating system has to trust.
- A promoted Bench is recorded as a recipe listing the seed, the apt and pip packages by name with versions where they are pinned, the network profile, and the granted directories as an approved upper bound.
- Reuse rebuilds instead of restoring: fresh base, reinstall the declared packages, re-apply the declared grants inside the bounds the recipe approved.
- The stored recipe is owned by the privileged supervisor, with the agent reduced to a client that requests resources and does not own the state.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A committed image travels between machines with whatever the workload fetched over its allowed network still on disk, so everyone who later reuses that image inherits the credential without being told it is there.
- constraint A rebuild is only as reproducible as its pins. Any package the recipe names without a version is decided by the upstream repository on the day the environment is regenerated.
- cost Every launch pays a full reinstall of the declared packages instead of an image pull, and the operator running the sandbox pays that cost each time.
- decision Anything the agent built inside the sandbox now has to leave as an output, because promotion keeps declarations and the rebuild will not bring a compiled artifact back.
The work the author wanted to keep was real. A Bench that installed a toolchain, compiled something and got a pipeline running, all of it gone when the disposable Bench went [1].
The debris around that work is specific. At the end of a task a Bench holds package manager caches, build artifacts in temporary directories, log files from failed attempts, a half-finished experiment that never went to production, and the failed apt install that came before the one that worked [4]. Commit the filesystem and every one of those becomes a layer you copy around. Six months later, according to the post, you cannot tell why any given file in that image exists [6]. "You are shipping garbage because the snapshot does not distinguish signal from noise," the author wrote [5].
Diffing two images does not recover the difference. A filesystem diff tells you a file appeared; it cannot tell you whether the file was ever supposed to be there [8].
The second failure is the one that leaves the machine. While a Bench runs, the workload may read a granted file or fetch an API token over its allowed network, and a snapshot bakes whatever was sitting on disk into a durable, reusable artifact [7]. The host was sealed in Part 1 to avoid opaque mutable state the system has to trust, and a committed Bench is a user-writable artifact standing in for a clean base [9]. "A saved Bench smuggles that trust right back in through the front door," the author wrote [10].
So promotion records the recipe instead of freezing bytes [11]. The recipe is short enough that the author reads it as a plain diff and confirms it before it is saved [13]. "It excludes opacity because the recipe is a plain text file that anyone can read and verify," the author wrote [17]. Debris is absent from the rebuild because debris was never recorded, and no secret is frozen because the recipe never held one [16].
Two conditions decide whether the rebuild actually gives you the same environment twice. The recipe lists apt and pip packages by name, with versions where the author pins them [12]. A name without a pin resolves to whatever the upstream repository is serving on the day of the rebuild, so reproducibility stops exactly where the pinning does [19]. The post does not report rebuild times or say what happens when a pinned version disappears upstream.
In my view this is the right default when what you value about the environment is what it declared it needed. A fresh install at every launch is the price of a recipe you can read before it is saved [15].
What to watch
- Whether later parts of the series publish the recipe format and a worked rebuild of a promoted Bench.
- Whether the confirm-before-save step stays a human reading a diff or becomes a signed policy gate.
- Whether a package cache is added to speed the reinstall, and what that does to the clean-every-time property.