Agent evals need environment capture, and the sub-second sandbox is the cheap half

The figure comes with a caveat: none of the supplied material states a sub-800ms cold start ^[16]. What Fast Sandbox does state is a core objective of millisecond-scale container startup latency for delay-sensitive work such as code sandbox execution ^[1], and the mechanism it names is warm runtime pools that reuse ready Fastlet Pods instead of creating a Kubernetes Pod for every sandbox ^[2]. Read any sub-second number as that project's design target, not as a measurement someone published.

The reason a startup figure belongs near eval methodology at all is arithmetic. ClawsBench spans 44 tasks and 33 scaffolding conditions ^[11][12], which is 1,452 runs in a full sweep ^[14]. If each run gets a fresh environment, an 800ms create costs about 19 minutes of the sweep and a ten-second create costs about four hours ^[15]. That gap decides whether per-run isolation is the default or the first thing dropped when the deadline moves.

Latency is the cheaper half. The completion-argument paper reports that in a replay with the agent's actions held fixed, endpoint and terminal labels differed for every delayed operation, and a delayed write changed the next run's score when service state persisted between runs but not after isolation or verified reset ^[7]. A quick Fastlet Pod bounds container creation; it says nothing about whether the Gmail or Drive mock inside it is back at its snapshot ^[11]. ClawsBench also lists multi-step sandbox escalation among its recurring unsafe patterns ^[17], which turns the runtime selection, plain container against gVisor or Kata, into a containment question rather than a throughput one ^[4].