Build1 publisher2 min readPublished
A missing gp3 StorageClass made Storm's init containers blame Nimbus for two days
Apache Storm ships no maintained Helm chart and no migration guide, so one team wrote its own templates for EKS. The first thing that broke was a StorageClass name, and the only symptom was an init container log line.
The Engineer · Build desk
What happened
- An Argo CD application for Apache Storm sat red for two days while init containers on the Supervisor pods looped on one line: waiting for nimbus ...:6627.
- Apache Storm has no maintained official Helm chart, and nothing credible published for migrating the cluster to Kubernetes.
- The team evaluated G-Research's gresearch/storm chart with its Bitnami ZooKeeper subchart and wrote a thin local chart instead.
- The ZooKeeper and Nimbus volume claims stayed Pending, so those StatefulSet pods never scheduled and Argo reported the app Degraded with the StatefulSets OutOfSync.
- Once the volumes bound, Nimbus exited with /docker-entrypoint.sh: exec: nimbus: not found, and Storm UI failed the same way for ui.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Because the adopting team wrote the templates, the bootstrap order of ZooKeeper, Nimbus and Supervisors is its own code to maintain, and when it misbehaves there is no upstream release to wait for.
- exposure When an init container gates on a dependency, a fault below it shows up as a message about the component being waited on, so the first log an operator reads points away from the broken object.
- decision Keeping the StorageClass inside the application chart ties a cluster-scoped resource to one release's lifecycle, and the next team copying this layout inherits that coupling with the templates.
- precedent Being first in the organisation makes this chart the reference others will copy, including its entrypoint convention for private images.
The Supervisor pods polled nimbus:6627 and reported exactly what they could see. The fault sat three layers below that port: the chart asked for a StorageClass named gp3, the cluster carried only the legacy in-tree gp2, so the ZooKeeper and Nimbus volume claims never bound and Nimbus never started listening [10][11]. The EBS CSI driver itself was healthy [10].
The dev.to writeup is blunt about the class of error: the init container does not say the StorageClass is wrong, it says upstream is not ready [17]. Before rollout the team wrote down what it was afraid of, and this was on the list: "we feared silent PVC Pending forever", the author wrote [12].
The two failures cost very different amounts of attention. The storage fault held the Argo application red for two days with the logs naming the wrong component, while the entrypoint fault printed its own missing command in the first line [18].
That second one is worth the config line. Copied Helm examples pass bare args like `nimbus` or `ui`, which is correct against the upstream Docker Hub images. The team's private ECR images run a `docker-entrypoint.sh` that execs the first argument as a command, so they need `storm nimbus` and `storm ui` [15]. The Supervisors and the LogViewer sidecar already used the `storm ...` form; Nimbus and UI did not [15]. "Official image docs do not apply when you own the entrypoint", the author wrote [16].
Writing a thin local chart instead of adopting gresearch/storm means the team owns the templates and has to maintain them [3]. One piece of that maintenance is already visible: the gp3 StorageClass lives inside the Storm chart, so a chart prune would delete a cluster-scoped resource. The team took that over standing up another Argo Application for a single consumer [9].
Whether this chain reproduces elsewhere turns on two conditions that have nothing to do with Storm. Your chart has to name a StorageClass, and your cluster has to not define it [19]. On a cluster that ships gp3 by default, phase one (ZooKeeper and Nimbus alone, to prove the control plane and that PVCs actually bind) looks like a formality; on this cluster it caught the fault that would have stalled every later phase [6]. No team in the org had put Storm on Kubernetes before, in any account or region, and Storm still runs on EC2 and autoscaling groups elsewhere [4].
What to watch
- Whether the local chart becomes the org's internal Storm reference or gresearch/storm is adopted later.
- Whether the gp3 StorageClass stays inside the Storm chart after the first accidental prune.
- Phases three to five, the LogViewer sidecar, the private ECR images and Storm UI on the shared private ALB ingress.