Build1 publisher2 min readPublished
Explaining Buildkite's incident takes the node vocabulary Kubernetes hides from service owners
Lorin Hochstein rates Buildkite's public write-up above GitHub's on detail, and the first thing his own retelling has to do is teach clusters, nodes and headroom, the layer a platform keeps out of sight.
The Engineer · Build desk
What happened
- Lorin Hochstein published his own reading of Buildkite's public incident write-up, days after writing about a GitHub Actions incident at the other CI-as-a-service provider.
- Retelling the failure mode in his own words required him to define Kubernetes clusters, nodes and pods first, noting that on Amazon's EKS he believes a node is an EC2 instance.
- He labels the unused part of a node headroom, a term he says is not explicitly a Kubernetes concept, meaning the additional compute jobs that node could still run.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The engineer paged first sees only the layer the platform exposes, so finishing the diagnosis needs someone with cluster access, and that handoff happens while the queue is still growing.
- decision Anyone alerting on pending pods has to split the alert, because a slow image pull and a cluster with no room to schedule need different responses and one threshold cannot serve both.
- precedent A write-up detailed enough for an outsider to reconstruct and republish raises what readers can reasonably ask of the next CI vendor's incident report.
Red, green and gray on Hochstein's example graph do different jobs. Red is the desired replica count the autoscaler computed. Green counts pods in the running state, which the Kubernetes docs define as bound to a node with all containers created and at least one still running [11]. Pending is the gray band, and the docs put two unlike delays in that single state: a pending pod "has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run", and that "includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network" [10]. A pending count on its own does not say which of the two you have [15].
Image download is bounded by the network [10]. Waiting to be scheduled is bounded by whether any node still has headroom [7]. The HorizontalPodAutoscaler increases or decreases the size of a replicaset in response to load [9], and a replicaset is a group of identical pods, added only because the service needs more compute [8]. Nothing in that loop creates a node. With every node full, a scale-out decision raises the desired count and the pending count by the same amount [16].
Hochstein is precise about who can see any of this. Service owners know pods, he wrote, "because that's what's exposed to you", while "those implementation details are deliberately hidden from the service owners" [6]. The engineer paged for a CI queue that has stopped draining is looking at pods.
For this failure to transfer to another cluster, several things have to hold. The scaling signal has to act on pod count and not on node count [9]. The node pool has to stay fixed while that happens. And the pods have to be large enough relative to the node that headroom runs out before the replica target is met [7]. Buildkite is a CI company [3], and Hochstein notes that a service consuming from a job queue sees the number of pending jobs vary at any one time [8].
He also notes that Buildkite's write-up says scale out, and that he uses scale up and scale out interchangeably, which is more disclosure than most glossaries offer [14].
Hochstein calls what the report describes "a fascinating failure mode" [17]. The text in hand ends at the autoscaling graph, before the node-level cause is named [18]. On the comparison with the GitHub Actions incident he had written about days earlier [1], he is unambiguous: Buildkite's "incident report contains a lot more detail than GitHub's" [2].
What to watch
- Whether Buildkite publishes node-level numbers, such as instance type and per-pod resource requests, that would let an outsider reproduce the failure.
- Whether the full Buildkite report ties its scale-out to a cluster autoscaler or to a fixed node pool.
- Whether GitHub adds node or cluster detail to its Actions incident report.