Build1 publisher3 min readPublished
Agent Substrate v0.0.8 gives a freed worker to whichever agent retries first
Agent Substrate v0.0.8 rejects work outright when a WorkerPool is full, and the pool's four settings and template include no priority field. Past a replica or GPU ceiling, a batch job with a tight retry loop can take the worker an incident responder needed.
The Engineer · Build desk
What happened
- A kubectl query against the live WorkerPool CRD in Agent Substrate v0.0.8 returns four settings and a pod template: ateomImage, replicas, sandboxClass, sandboxConfigName and template.
- ActorTemplate, the object that describes an agent, lists containers, pauseImage, sandboxClass, snapshotsConfig, volumes and workerSelector, with no field that ranks one agent above another.
- When the pool is full, Substrate rejects the request with "substrate worker pool has no free workers" and keeps no queue of its own.
- In the author's lab, a killed session left a ghost actor holding a worker while the pool looked idle, and only a restart of the pool deployment freed it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Once a replica, node pool, GPU or rate-limit ceiling stops the scaling, arrival timing alone decides which agent waits for a worker.
- exposure Incident-response agents with polite backoff are the likeliest to lose workers during contention, and on-call staff will see them fail while the batch job that took the slot looks healthy.
- decision Teams that need priorities on Substrate v0.0.8 have to build admission control in their callers or in front of the pool, because the runtime schema has nothing to configure.
- precedent Any priority field added upstream will need aging or fairness alongside it, or low-priority agents will stop running whenever the system is busy.
The author's own autoscaler shows where the gap sits. The policy, shipped in the author's visualizer, sets `target = busy + queued`, clamped to [2, 8]. It scales up straight to target when `queued > 0` for two samples (6s), with an 8s cooldown. It scales down to the peak demand of the last 30 seconds after six samples (about 18s), with a 20s cooldown [5]. Both windows put the sample interval at 3 seconds [1]. The input is a count. "The policy knows how much work is waiting and knows nothing at all about whose work it is," the author wrote [11].
The queued figure is itself an estimate. Substrate keeps no queue of its own [3]. Waiting work exists only inside each caller's retry loop, and the post says any autoscaler on top is measuring demand secondhand [4]. Once `busy + queued` passes 8, the clamp stops the target from rising [2]. That cap is a replica ceiling. The author lists it alongside node pool limits, GPU budgets and model provider rate limits as the points where adding capacity stops being available [6].
Above the cap, a free worker goes to whichever caller retries into the window when it opens [7]. In the post's example, an overnight enrichment agent with a tight retry loop beats an incident responder with a polite backoff every time, and the responder is the agent that looks broken [7]. The winner keeps the slot. A low-priority actor holds it for the length of its session, and sessions with an LLM in them are long and variable [8]. "When the holder of a contended resource cannot be preempted, priority is not a policy, it is a hope about arrival order," the author wrote [12].
A priority field on its own would swap one failure for another. Serve the highest priority first with no aging or fairness rule, and according to the post the lowest-priority agent runs when the system is quiet and never when it is busy [10]. The author describes the whole problem as "the part I do not think anyone has solved, including me." [13]
The evidence deserves credit. The post says the demo repo and every command in it were run before publishing [15]. The schema check is one `kubectl get crd workerpools.ate.dev` piped through a Python one-liner, and anyone can rerun it against their own cluster [1]. It is also one runtime, at v0.0.8, on a kind cluster [1]. A schema with two leading zeros in its version is allowed to be missing things. The wider line, "Every capacity knob in an agent runtime today is an efficiency knob," is the author's claim about the category [14], and the post tests it against Substrate alone. For the Substrate finding to carry to another cluster, the WorkerPool and ActorTemplate schemas must still lack a priority or preemption field, and the pool must actually reach its ceiling. Below the ceiling, the author calls scaling a good answer to contention [6].
What to watch
- Whether a later Agent Substrate release adds a priority, preemption or queueing field to the WorkerPool or ActorTemplate schema.
- Whether the ghost-actor case, where a killed session pins a worker until the pool deployment restarts, gets an upstream fix.
- Whether the author's notes in the kagent-substrate-demo repo move from the problem to an admission-control design with aging or fairness.