Ai2 swaps priority scheduling for GPU time budgets after every job ended up marked HIGH

A priority field orders work only while some jobs sit below the top level. At Ai2, priority inflation ran until 100% of scheduled workloads used HIGH, and the lower levels got no GPU time at all ^[8]. At that point the priority field ranked nothing. The preemption rule added its own incentive. A team's concurrency cap covered only jobs protected from preemption, and preemptible jobs could exceed it on idle GPUs ^[6].

The first fixes stayed inside the old model. The team tightened control over how priorities were set, then worked around its own scheduler by assigning GPU monopolies to important projects ^[10]. Its engineers now call the period a "tragedy of the commons", in which researchers maximizing their own results on a shared resource produced a worse global result ^[13]. Monopolies are the classic answer to a commons: privatize it ^[13]. The available text of the post ends mid-sentence in that passage ^[14]. It does not describe how budgets are sized or what the time-slicing contract requires, and it includes no before-and-after figures.

Gaming of this kind is old. In the 2011 paper that introduced Dominant Resource Fairness, Ghodsi et al. recount a search company that gave dedicated machines only to jobs whose users could guarantee high utilization. The company, they wrote, soon discovered "users would sprinkle their code with infinite loops to artificially inflate utilization levels" ^[12]. Ai2's version was a no-op workload parked on a GPU for later ^[7].

Ai2 ranks its infrastructure goals as a pyramid of availability, occupancy, impact and utilization ^[11]. Occupancy is the share of available time assigned to a workload ^[11]. A parked no-op job counts toward it ^[7]. The budget redesign targets impact, defined as how often the most valuable workloads are chosen to receive resources ^[11].

For another lab, the question is which of these failures it shares. Ai2 has outstanding requests for 2-3x the GPUs it has ^[5]. At that ratio, between half and two-thirds of requested capacity waits at any moment under any scheduler ^[15]. Budgets decide whose work waits. They do not address the cause Ai2 gave for squatting, which was that debugging workloads could not launch with low enough latency ^[7]. I'd expect a lab that copies the budgets without a fast interactive launch path to see parked jobs come back.

The on-call cost is the strongest case for copying. Engineers spent a majority of their ticket response time negotiating the shutdown of non-preemptable workloads on hosts with known maintenance problems ^[9]. That time traces directly to letting workloads opt out of preemption ^[6]. A lab weighing this design has to settle that rule before it sets a single budget.

Scale and governance matter too. About 150 researchers share thousands of H100, B200 and B300 GPUs in clusters of 88 to 1,024 ^[3][4]. Ai2 describes the new allocation as a transparent administrative budgeting process ^[2]. Someone has to hold the authority to run it.