KEDA ties Metrics Server readiness to the gRPC link behind every HPA query

The Metrics Server does no metric work of its own. In KEDA, the CNCF event-driven autoscaler, the keda-operator talks to scalers and computes metrics ^[13][4]. The Metrics Server is the custom metrics API server the HPA queries. It forwards each request to the operator over gRPC and relays the answer ^[4]. "That gRPC connection is the Metrics Server's entire reason to exist," wrote the developer who hit the bug and submitted the first version of the fix, in a post on dev.to ^[14][17].

So a probe that reported Ready while that link was down was checking the wrong thing ^[1]. According to the post, the HPA stops receiving scale signals in that state while the replica still looks healthy ^[2]. "A crash is loud," the author wrote ^[15]. This failure sends debugging the wrong way, the post says, because the first thing anyone checks is whether the pods are ready, and they are ^[18].

Version one used state gRPC already tracks. grpc.ClientConn.GetState() returns Idle, Connecting, Ready, TransientFailure or Shutdown ^[7]. It reports state and never starts a connection, so a probe can call it with no side effects ^[7]. The patch added an accessor on the metrics-service client and a readyz check in the adapter that failed unless the state was Ready ^[8]. When the operator link dropped, Kubernetes pulled the replica from the APIService endpoints and the HPA stopped sending it queries ^[8].

The review comment is the more instructive part of this change. A maintainer spotted a failure that exists only because version one worked ^[9]. The post sets out the cycle like this:

1. The operator connection goes bad and the probe reports NotReady ^[11]. 2. Kubernetes removes the replica from the APIService endpoints, and no RPCs flow through that client ^[9]. 3. With no RPCs, gRPC parks the connection in Idle. GRPC_CLIENT_IDLE_TIMEOUT_MS defaults to 30 minutes ^[10]. 4. The Idle connection waits for an RPC. RPCs come back only after the probe reports Ready, and Ready needs the connection awake ^[11].

"An idle gRPC connection does not reconnect on its own. It waits for someone to use it," the author wrote ^[16]. The post says a replica caught this way can stay NotReady forever ^[11].

Kubernetes did exactly what the probe told it to in both bugs ^[6][9]. According to the post, the merged change has the readiness check nudge an Idle connection to reconnect ^[12]. The check gives up the property GetState() was picked for, because it can now start a connection as well as observe one ^[19]. We think that is the right tradeoff for this component. Once the replica is out of the endpoints, no RPC will touch that client, and the readiness check is the one code path left that can ^[20].