Build1 publisher3 min readPublished
Your Nodes Are Not Idle, They Are Reserved: The Requests Trap In Karpenter Consolidation
An EKS cluster refused to consolidate while dashboards showed nodes at 10% CPU. The scheduler was reading resource requests, not utilization, and one workload requested 250x what it used.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- Engineering Build Notes #2, published on dev.to, documents how Kubernetes resource requests became a scheduling constraint in an Amazon EKS cluster.
- The worker nodes appeared to have more capacity than the workloads were using, and the obvious optimization would have been to make them smaller.
- One node was consistently running at approximately 98-100% CPU utilization while memory utilization remained around 12-15%; the bottleneck was CPU, not memory.
- Instead of downsizing the workers, the author increased the node size from t3.medium to t3.large, removing the CPU constraint.
- The cluster was running with Karpenter managing worker-node capacity.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
An engineer documenting an Amazon EKS cluster on dev.to found nodes that would not leave the cluster even though their observed CPU and memory looked low, and traced the cause to Kubernetes resource requests rather than to Karpenter itself [1][6]. That matters because the two numbers most operators watch on a dashboard, CPU and memory utilization, are not the numbers the scheduler is allocating against [11].
The sequence is worth following because the first instinct was also wrong. The worker nodes looked over-provisioned relative to what the workloads were using, and the obvious move was to shrink them [2]. The metrics said otherwise: one node sat at roughly 98-100% CPU while memory stayed around 12-15% [3]. The constraint was compute, not memory, so the author went up rather than down, from t3.medium to t3.large [4].
More headroom did not produce better packing. With Karpenter managing worker-node capacity [5], nodes persisted despite low observed usage, and consolidation events were visible during the investigation [6][7]. Rather than reach for the autoscaler's knobs, the author compared actual consumption with configured requests, workload by workload [19].
One workload made the gap legible. Observed usage was about 1m CPU and 17-23Mi of memory, against configured requests of 250m CPU and 250Mi of memory [8][9]. That is roughly 250 times the CPU it was using and about 12 times the memory, reserved on the node whether or not the pod ever touched it [10]. Multiply that across several services with similarly inflated requests [19] and a node reporting 10% CPU and 25% memory on a dashboard [12] can be effectively full from the scheduler's point of view, which is what produces less efficient packing [13].
The discipline in this account is the part worth copying. The available Karpenter changes were all tempting and all available: more aggressive consolidation, different disruption settings, NodePool edits, forcing nodes out [16]. The author's argument is that each of those changes the infrastructure layer before the workload layer is understood, and that if workload requirements do not reflect workload behaviour, tuning the autoscaler hides the problem instead of fixing it [16][18]. The dependency chain runs from workload requests through the scheduler to placement, then to consolidation opportunities, and only then to Karpenter's decisions [17]. The author is explicit that requests are not the only input; Karpenter also evaluates several scheduling and disruption constraints [14]. Requests were simply the one constraint that could be controlled directly [15].
The remediation was not a blanket cut. One class of workload moved from requests of 100m CPU and 512Mi memory, with limits of 800m and 1Gi, to requests of 25m and 64Mi with limits of 300m and 256Mi [21], a 4x reduction in requested CPU and 8x in requested memory [22]. The stated goal was requests that represent the workload while keeping reasonable operational headroom, not the smallest possible numbers, and each workload was reviewed individually [20][23].
What to watch: whether request cuts of that size hold under real peaks, since requests also govern eviction ordering and burst behaviour, and the reduced limits in the example are a separate bet from the reduced requests [21]. Watch, too, for the second-order effect on cost, which is what started this series [24]. If your consolidation stalls, compare requested against used capacity per node before touching the NodePool.