A misconfigured autoscaling policy watched the host service, not the Istio sidecar that saturated

An autoscaling policy comes down to two choices: which metrics track load, and how resources are added or removed as those metrics move ^[14]. The defect sat in the first choice. Reading GitHub's paragraph, surfingcomplexity.blog concludes the policy for the impacted service used load metrics that took into account only load on the service itself, and not on the Istio sidecar ^[16]. GitHub's own wording is that the sidecar failed to auto scale "because of a misconfigured policy that watched host service but not sidecar limits" ^[4].

That blind spot is ordinary. CPU utilization is the common autoscaling metric, and a service can be saturated while CPU is low ^[17]. The blog's worked example is thread-per-request with a thread pool: downstream latency rises, every thread ends up blocked on I/O, the service badly needs more pods, and CPU reads low because the threads are waiting ^[18]. Slack hit that in 2021 and responded by scaling up rapidly on the number of threads ^[18][19].

So for watching the sidecar limit as well to transfer to your stack, two things have to be true. The sidecar's concurrency headroom has to be visible as a metric your policy can read, and someone has to drive it to that limit deliberately. surfingcomplexity.blog argues that every autoscaling policy is effectively bespoke, which leaves the owning team running an operational control system with custom parameters "that can really only be checked via load testing", and those owners are "almost certainly not autoscaling experts" ^[20]. The post asks the follow-up out loud: "Are you doing load testing on all your services?" ^[21]

Two retry loops ran during the incident, and the write-up treats them separately: optimistic retry logic against the load balancers, and client retry against the Copilot Token Service ^[9]. The first overloaded internal load balancers ^[6]. The second kept Copilot authentication failing after the rest recovered, because a failed token operation could generate many extra requests and enter a retry loop ^[7]. GitHub attributes the client side to a latent bug in Visual Studio Code, triggered by delayed replies to a single internal endpoint, and puts the amplification at approximately 10x ^[8].

The blog calls the failure mode yet another example of saturation, a topic it gave a talk on a month before the incident ^[25]. It also warns against stopping at the policy, invoking David Woods's component substitution fallacy: the belief that reliability improves by finding and fixing defective components ^[22]. Its argument is that your system is full of latent component defects right now and is not constantly failing over, so the defect alone was not sufficient ^[23]. The factors it treats as interacting here include changing traffic patterns with scrapers among them, the autoscaling policy, sidecar saturation, retry logic, HAProxy saturation and authentication traffic ^[24].

One thing worth noting for anyone reconstructing the diagnosis: auth broke without auth traffic rising, and without a recent change to the auth system, which the blog says is the first place it would have looked ^[12]. The write-up does not cover the responders' diagnostic work ^[11]. The blog calls the write-up surprisingly quick and says it would have preferred a preliminary version now and a detailed one in a couple of weeks ^[2]. During the incident, the status page alternated between "we have identified the problem" and "we are experiencing issues" ^[13].