Skip to content

Build1 publisher3 min readPublished

Locking the EKS API to one IP left the worker node with no route to the control plane

A k3s workload moved to Amazon EKS without an application code change, and the three things that had to be fixed were the control plane endpoint policy, the instance type AWS would allow, and the credential doing the work.

The Engineer · Build desk

Illustration accompanying Locking the EKS API to one IP left the worker node with no route to the control plane

What happened

  • The same Node.js application, the same image selected by immutable digest and the same Helm chart ran on Amazon EKS after moving off a home k3s server, with no application code change.
  • The author's interactive AWS login stopped refreshing reliably during the long create-and-check cycle, and a temporary role carried the remaining setup and teardown work.
  • Argo CD, reading Git from inside the cluster, brought up two healthy replicas and corrected a manual scaling change in 5 seconds.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost The stated goal was an always on Kubernetes environment without an always on EKS bill, and the write-up publishes no dollar figure for the temporary cluster, so the saving anyone copying this pattern is chasing stays unpriced.
  • exposure A personal interactive session in the create path makes the cluster's lifecycle depend on one human's credential staying refreshed for the length of a Terraform apply.
  • capability One shared Helm chart can target both a private ClusterIP at home and an AWS load balancer on EKS without forking, because the environment-specific choice sits in a small values file.

The control plane came up active before anything looked wrong. Private access was off and the public endpoint allowed one address, the author's own IP, so the managed node had no path to the API it needed in order to register, according to the dev.to write-up [9][11]. The endpoint was reachable from exactly one laptop, and kubelet does not run on laptops. Enabling the private endpoint for traffic inside AWS, while keeping public administrative access restricted, was the fix [12].

The second break came out of AWS's own catalogue. The plan called for a t3.medium, and the account's eligible configuration rejected it [13]. The author queried the eligible AMD64 types and took one c7i-flex.large with 2 vCPUs and 4 GiB of memory; the same container and the same Helm release ran afterwards [13][20]. "Kubernetes can preserve the workload contract, but the cloud provider still controls which underlying compute choices are available," the author wrote [14].

Third, the operator credential. The interactive login stopped refreshing reliably during the long create-and-check cycle, so the remaining setup and teardown ran under a temporary role that was removed during cleanup [15]. "A repeatable EKS workflow needs a stable, non-personal AWS role with only the permissions the workflow requires," the author wrote [16].

What needed no change was the part of the pipeline that never held a cluster credential. CI tested the code, built an immutable image and proposed a digest change in Git, and it could not deploy to either cluster [7]. Argo CD ran inside the cluster and read Git from there [5]. The health, readiness and version endpoints came across as they were [6].

The timings say where the work actually went. Creating the control plane took 5 minutes 51 seconds, which is 351 seconds, roughly seventy times the 5 seconds Argo CD needed to undo a manual scaling change [2][3][1]. The CI checks on the pull request that merged the EKS configuration finished in 13 seconds [17].

Two limits bound how far this transfers. One managed worker held the bill down and left the workload nowhere to run if that node failed, which the author accepted for a short experiment [10]. The two public subnets sat in separate Availability Zones, but one node sits in one subnet, so the zone spread existed in the network layout and not in the compute [9][2]. "This was a portability test, not a production EKS design," the author wrote [18]. The run therefore says nothing about how the same chart and the same Argo CD contract behave across a node failure, a second Availability Zone with real capacity in it, or a managed node group that has to schedule more than one replica per host.

For the mechanism to carry to a production cluster, three things would have to hold: the endpoint policy would need to be an explicit Terraform input [9], the instance type would have to be selected from what the account can actually launch [13], and the role running the apply would have to outlive one person's session [15]. Kubernetes was the portable layer here, and everything under it and around it was an AWS decision [4].

What to watch

  • Whether the same chart and Argo CD contract hold on a multi-node managed node group with real capacity in a second Availability Zone.
  • Whether the temporary operator role is replaced by a scoped, non-personal AWS role created in the same Terraform run, and what permissions it ends up with.
  • A published cost figure for the temporary EKS environment, which would let the 'no always on bill' premise be checked.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories