Build1 publisher3 min readPublished
Thirty Horizon queues pin thirty idle workers before the first job arrives
Laravel Horizon sets worker allocation with one key per supervisor. Switch balancing on and minProcesses keeps a booted worker watching every queue you list; switch it off and the list becomes a strict priority order a flooded queue can own.
The Engineer · Build desk
What happened
- Horizon makes each supervisor pick a balancing mode. 'auto' or 'simple' give every queue its own worker pool, at the cost of a permanent floor of resident workers that grows with the queue count.
- The auto balancer never scales a queue below minProcesses, which defaults to 1, so every queue named in the supervisor's list keeps a booted worker watching it even while it is empty.
- With balance set to false there is one shared pool and no idle minimum. In the article's example, fifty thousand jobs on high take every worker while default and low get none until high drains.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Under auto balancing, per-queue observability costs RAM: every queue a team adds for its own dashboard row and pause switch is another resident PHP process, idle most of the time.
- constraint In the published example the balancer cannot hand a flooded queue more than about a quarter of the pool. The only way to give it more is raising the peak process count for the whole box.
- decision Since balance is set per supervisor, the real design choice is how to group queues into supervisors. That grouping decides which queues can starve each other and how many pools you have to size.
- capability Weighted polling would let a team keep thirty queues for pausing and draining without either the resident floor or the strict order, provided the pickup ratios hold while the top queue is flooded.
The per-queue minimum consumes most of a supervisor's process cap. Horizon's `minProcesses` defaults to `1`, which guarantees at least one worker is always watching each queue in a supervisor's list [3]. In the example the article publishes, that list holds thirty queues and `maxProcesses` is `40` [4], so thirty processes are pinned and the auto balancer has ten left for whichever queue is actually busy [5]. The flooded queue therefore tops out at eleven processes, its own minimum plus the ten spare [1]. Three quarters of the ceiling is committed before a single job is dispatched [3].
Those thirty are booted, resident and idle most of the time, because most queues are empty most of the time. Each PHP worker is a full framework boot holding tens of megabytes of RAM whether or not it ever picks up a job, according to the article [6]. Taking "tens" at face value, thirty of them is roughly 300 MB at the low end and close to 3 GB at the high end [2].
Turning balancing off removes the floor and replaces per-queue isolation with list order. Every worker in the single pool listens to every queue and checks them in the order you listed them [7]. A worker looks at `default` only when `high` is empty, and at `low` only when both are empty [8]. Push fifty thousand jobs onto `high` and the other two get no workers for as long as `high` stays non-empty [9]. Under light load you will not notice, per the write-up; under sustained load the order decides who gets served at all [10].
The reason a healthy app has thirty queues is that a queue is the smallest thing Horizon will show you, and Laravel's `queue:pause` and `queue:clear` each act on one queue at a time [11]. Splitting `exports` out is how you empty a backlog without touching `payments` [11]. Each split adds either one more resident worker or one more rung on the priority ladder, and the article is explicit that both costs scale with the queue count [12].
Grouping is the one lever the config itself exposes. `balance` sits inside a supervisor block next to that supervisor's own queue list [4], so two supervisors can run two unbalanced pools over two shorter lists [4]. Starvation then happens inside a group, and the bill is one sized pool per group [4]. Capacity sized for a quiet group stays in that group's pool when the busy one needs it [4].
The fix the article proposes is `queueWeights` in Skyline. It keeps the shared pool of unbalanced mode but checks queues in proportion to their weights, 3:2:1, instead of in strict order, so important work gets most of the pickups and no queue drops to zero [13]. Skyline also pauses and resumes a single queue from the dashboard instead of the whole supervisor [14]. The post is written to make that case [15], and it does not include measurements for either mode [16]. Weighted polling addresses the starvation the article describes. Whether a 3:2:1 split survives fifty thousand jobs on the top queue depends on whether each worker honours the weights independently or the pool honours them only in aggregate.
What to watch
- Whether Horizon adds a weighted strategy or changes the minProcesses default upstream. Either change would remove the reason to run a third-party dashboard for this.
- A measured per-worker RSS figure and a pickup distribution for weighted queues while the top queue is flooded.
- Whether per-queue pause and resume from the dashboard lands in Horizon itself rather than only in Skyline.