Build1 distinct publisher3 min readPublished
Putting every manifest in git makes a dead datastore survivable, but only if reconciliation arrives in order, and the order is set by the fact that Envoy Gateway's CRDs ship inside the controller's own Helm chart.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Flux reconciles a Kustomization by pushing objects at the API server, and a GatewayClass is a valid object only once the CRD that defines that kind is registered. Envoy Gateway delivers those CRDs inside its Helm chart [5], so the type definition shows up as a side effect of installing the controller. Skip the ordering and, in the author's account, Flux runs ahead and tries to create a GatewayClass in a cluster that has never heard of GatewayClasses [6]. The three chained Kustomizations plus `wait: true` encode the sequence instead: controllers, wait until healthy, then configs, then apps [7].
That transfers only if your CRDs are packaged the same way. The dependency edge exists because the chart is the delivery mechanism for the schema [5]. Where CRDs are applied as a separate step, the graph has a different shape, and copying someone else's `dependsOn` chain gives you serialization you did not need.
Serialization is the bill. On this layout the controllers Kustomization holds both Longhorn and Envoy Gateway HelmReleases [3], and the apps Kustomization sits downstream of it [4], so a storage controller that never reports healthy is now also the reason Airflow never gets applied [7]. For a single-node homelab rebuilding after a power cut [1] that is the right trade. For a cluster where app teams ship independently of the platform, one health check upstream of everything is a queue.
The secrets failure is the same class of bug with no `dependsOn` available to fix it. The Airflow HelmRelease reads its admin password through `valuesFrom` with `optional: false`, so no secret means no install [13]. Bootstrapping first and creating secrets second gave Flux three retries, an exhausted budget, and a release parked in a failed state [14]. Then remediation reported `missing target release for rollback: cannot remediate failed release`, because a first install that never succeeded leaves nothing to roll back to [15]. Recovery needs `--force` or a suspend/resume, and the stated lesson is to create the secrets before bootstrap, since the cluster exists as soon as MicroK8s is up and Flux does not have to be watching for `kubectl create secret` to work [16]. That last sentence is the kind you only write after it costs you an evening.
Version pinning belongs in the same argument. Every HelmRelease started at `version: "1.x"` [17], and chart 1.22.0 of Airflow carries `appVersion: 3.2.2` [19], which is a useful reminder that a chart range constrains the chart's numbering space and not the application's. The repo now names 1.22.0, 1.12.1 and v1.9.1 [18], which moves upgrade timing out of the reconcile loop and into a commit [17].
Three of the documented stumbles are ordering errors rather than value errors: CRDs before their consumers, secrets before the release that mounts them, and the directory before the kustomization entry that references it, the last of which surfaced as a reconcile hanging on `context deadline exceeded` with no pods appearing [1][20]. Git holds the desired state; the order that state has to arrive in lives in `dependsOn` edges and in a script you run before Flux ever starts watching [4][16].
Ranked by verification strength, evidence, and original report placement.
A power cut exposed that the cluster's entire memory lived in one SQLite-flavored database, on one laptop, bound to one Wi-Fi address, guarded by one aging battery, which the author describes as four single points of failure.
The author's stated fix is not making the datastore unkillable but making it unimportant: every manifest lives in git and something reconciles the cluster against git continuously, so a dead datastore becomes a reboot with extra steps.
Repo layout: clusters/homelab/ holds flux-system (written by Flux at bootstrap), infrastructure.yaml and apps.yaml; infrastructure/controllers holds Longhorn and Envoy Gateway HelmReleases; infrastructure/configs holds GatewayClass, Gateway and StorageClasses; apps/homelab/airflow holds Airflow.
Three Flux Kustomizations are chained via dependsOn: infra-controllers, then infra-configs, then apps.
The author's GatewayClass cannot exist until Envoy Gateway's CRDs exist, and those CRDs arrive with the controller's Helm chart.
Without dependsOn, according to the author, Flux sprints ahead and tries to create a GatewayClass into a cluster that has never heard of GatewayClasses, and fails.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 30, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Flux moves GitOps' source of truth into registries you own, and mirroring becomes the prerequisite1 distinct publisher
build
A 30-to-45-second timeout change, four approvals, no merge: the cost of a two-person gate1 distinct publisher
build
Your scanner finds it in seconds; the average fix now takes 252 days1 distinct publisher
build
Partition, not consolidation: what a 43-minute Jenkins queue actually cost1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One operator's logbook, quoted closely
The sturdiest material here is what the author could not have smoothed over: the pasted "missing target release for rollback" text, three exact chart pins, the dirty-state dot in a shell prompt. Those read as transcription rather than recollection. The softest is the part readers are most likely to copy straight into a GitHub settings page — the fine-grained token scope list, offered as a "docs-diving summary" with nothing linked to verify it. One publisher, one homelab, no second pair of eyes anywhere in this reporting.
A single homelab, rebuilt once
Count the installations in this story and you get one: MicroK8s on a laptop plus a Dell, reconciling Longhorn, Envoy Gateway and Airflow from one repository. The only outcome measured against it is the author's own twenty-minute rebuild, most of it image pulls. Nobody else's setup, no upstream usage figures, no third party reporting the same ordering behaviour.
Undersold as a war story
Claims that arrive with their own caveats rarely need discounting. The author volunteers that Flux restores declarations and knows nothing about Postgres data, DAG run history or Longhorn volume contents, and says plainly that the recovery plan only works alongside the previous episode's nightly snapshots. Meanwhile the genuinely portable finding — that a controller shipping its CRDs inside its own Helm chart dictates the reconciliation order for everything downstream — is filed under homelab mishap rather than stated as the rule it is.
Series momentum, no vendor money
This is episode six of a running dev.to series, and failure is the product: an instalment where nothing broke would be a shorter post with fewer readers. That pulls toward a tidy four-mistakes-and-a-lesson arc — but it pulls in the same direction as candour, and the elements hardest to tidy, the error strings and the version numbers, are exactly the ones present. No vendor, sponsor or product sits behind any of the four projects being recommended.
Sure what happened, unsure how far it travels
We can be reasonably confident about what was configured and what broke, because the details are narrow, dated and quoted. Confidence drops on reach: two nodes of MicroK8s in a spare room are a thin basis for the token-scope prescription, and nothing in this reporting tests whether the same sequencing behaviour appears under a managed control plane or with a different gateway controller.