Pulumi restores its Neo agent's files from Kopia snapshots after Git rebuilds fell short

Picture a platform engineer who gives Neo a provider upgrade in the morning. It is the same job Pulumi uses as its example: bump the version across a repository, regenerate the SDK code, run previews and tests, open a pull request ^[2]. Halfway through, the runtime restarts, and the agent picks the task back up from its replayed history ^[2]. In Pulumi's words, it "can wake up remembering the plan but missing the room it was working in" ^[18].

Pulumi's post starts from the point that a task is also "a working directory that has to survive long enough for the agent to keep making progress" ^[20]. What Neo does all day looks like an engineer's day: it clones repositories, edits files, installs dependencies, runs previews and leaves intermediate work behind ^[1]. Its workspace holds generated files, scratch files, local commits, package-manager state, tool output and sometimes files outside any repository ^[19]. The first recovery design saved only the Git part of that. For simple hosted cases it worked, and it kept stored state small ^[16].

The Git rebuild also needed the outside world to stay put. A resume required the remote to exist, the credentials to work, the saved revision to be available and the patch to apply cleanly ^[17]. Tasks failed with clone errors when a repository was deleted after the task started ^[7]. Large diffs outgrew the payload limit for persisted task state ^[8]. The post lists these failures by type and does not say how often each occurred.

Pulumi sums up the redesign in two sentences: "Replay events to restore what Neo remembers. Restore a snapshot to recover what Neo changed." ^[11] The snapshots cover Neo's hosted runtime. When a user runs `pulumi neo` on their own machine, the file and shell tools execute locally ^[4].

Strip out the continuity language and what Pulumi built is a directory backup. The harder decision was which backup. Pulumi wrote that the deciding question was "what are we willing to make every Pulumi Cloud and self-hosted customer operate so Neo can resume a task?" ^[13] A dedicated Git server failed that test for a reason close to the one that sank the Git rebuild: Neo workspaces often include repositories, but the workspace itself is not one ^[15].

For any agent whose tasks can outlive the process running them, I'd snapshot the whole working directory and stop reconstructing it from Git. The tradeoff is one more encrypted store to run and to test restores from ^[10]. A team that ships to its own hosted runtime can absorb that. A vendor whose customers self-host has to fit it into storage those customers already operate, as Pulumi did ^[14].

Sort each agent task on two tests: whether the step after a restart reads anything from disk that is not in a pushed commit, and whether the task is cheap to rerun from the start.

- All state in pushed commits, cheap rerun: replay the conversation and re-clone. Pulumi's Git design worked in this corner ^[16]. - All state in pushed commits, expensive rerun: Git reconstruction holds until a repository is deleted or a credential stops working ^[17]. - State off-commit, cheap rerun: restart from zero and accept the lost time. - State off-commit, expensive rerun: snapshot the directory, including the dependency caches and build output a backup policy would usually skip ^[9].