Skip to content

Build1 publisher3 min readPublished

One shared timestamp decides which priority job gets through an open circuit breaker

Stopping all 15 guarded jobs uniformly cut a posting job to a 43% execution rate, so the guard now lets a job flagged --priority send one real request every 1800 seconds, from a budget every priority job shares.

The Engineer · Build desk

Illustration accompanying One shared timestamp decides which priority job gets through an open circuit breaker

What happened

  • When claude-quota-guard.py detects the quota limit it returns exit code 75 until open_until and stops all 15 guarded jobs uniformly, regardless of which job consumed the quota.
  • Cumulative launchd.log figures put the posting job xpilot.autopost at 231 runs against 308 skips, a 43% execution rate, below threadspilot.engage at 64%.
  • open_until is set from the reset time parsed from the limit message or from a 6-hour cooldown, so the circuit stays shut on that schedule even when the quota comes back sooner.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Flagging a second job --priority buys no extra attempts: all priority jobs draw on the same 12 slots a 6-hour cooldown allows, so they now compete with each other rather than with the engagement jobs.
  • capability A probe that succeeds lets the guard learn the quota is back before open_until arrives, so the cooldown becomes an upper bound on the outage instead of a fixed wait.
  • exposure The probe path depends on one environment variable surviving into the child process; lose it and the slot is spent on a call the inner guard rejects with 75 before any request is made.
  • decision Choosing which of the 15 guarded jobs carries --priority is now the scheduling call that decides whose work survives a quota exhaustion caused by something else.

Inside run_job the order matters. The function calls circuit_status() first. If the circuit is open, two things have to hold before the job command runs: the caller passed --priority, and claim_priority_probe() returned true [5]. Miss either and the guard prints CLAUDE_QUOTA_JOB_SKIPPED to stderr and returns 0 [5]. The exit status is success, so anything watching return codes cannot tell a skipped run from a completed one, and the stderr marker is the only record; quota-catchup.py reads those markers when it re-runs the backlog [22][15].

claim_priority_probe() does one thing. It opens locked_state(), reads last_priority_probe, and returns False if less than PRIORITY_PROBE_INTERVAL has passed; otherwise it writes the current time and returns True [7]. The default interval is 1800 seconds [8]. That timestamp is per circuit, not per job, and it lives in a JSON state file guarded by fcntl.flock [9], which for one shared integer is the right amount of machinery. The docstring gives the reason for sharing it: repeated calls while the limit is genuinely in force do not bring the quota back, so the number of attempts is capped on purpose [10].

The probe budget is therefore fixed. The fallback cooldown is six hours [4], and at 1800 seconds a slot, that is 12 attempts across the whole open window, shared by every job carrying --priority [19]. Two priority jobs alternating get six each. The timestamp is written at the moment the probe is claimed, before subprocess.run launches the child, so a child that exits without issuing a request has still spent the slot [21].

The request itself happens in that child, and the guard is in its PATH. When the job calls the real claude binary, the call goes back through the guard and circuit_status() runs a second time inside the child [13]. The parent has to tell the child it is a probe, which it does by setting CLAUDE_QUOTA_PRIORITY_PROBE=1 only when probe_claimed is true [11]. The inline comment at lines 491-498 says that without the flag the circuit returns 75 immediately [12]. A probe that loses the variable spends its slot and produces a 75 without a request leaving the machine [24].

That 43% is one machine's cumulative launchd.log tally, over one quota account and one job mix [2]. 231 runs plus 308 skips is 539 scheduled firings, and 231 divided by 539 is 42.9% [18]. Two conditions have to hold for the same starvation to appear in another fleet: the breaker trips on a resource shared across jobs, and the jobs consuming most of that resource fire most often. Both hold here, and the comment at lines 18-24 attributes the collateral damage to the reply and engagement jobs, which account for most of the consumption [3].

Posting got the flag for a reason unrelated to throughput. The author wrote that it is a job where "if you miss the time slot, it never gets filled" [17], and that a successful attempt doubles as early detection of quota recovery, where the previous behaviour was to wait out open_until blind [14].

The post's introduction names two pitfalls and says one of them silently dropped a daily job for four days, 2026-09-13 to 16 [16]. The available text stops at the definition of run_claude, before either is explained [23]. Two hazards are visible in the code as shown: the shared timestamp, which lets one priority job consume the interval for all of them [9], and the environment flag, which is the only thing distinguishing a probe from an immediate 75 inside the child [12].

What to watch

  • Whether last_priority_probe gets keyed by job label, which would end the contention between priority jobs.
  • A fresh launchd.log tally putting xpilot.autopost above 43% would be the first evidence the probe path is firing.
  • Whether a successful probe actually shortens open_until or only logs that quota is back.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories