Build1 distinct publisher2 min readPublished
A clean sync, a green dashboard, and ninety seconds later real 500s. One team's rewrite moves the health verdict out of Argo CD and into Rollouts, gated on Prometheus error rate and p99.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The health model is not broken, it is narrow. Argo CD asks whether the pods of a plain Deployment are running and ready, and reports that back as the application's health [2]. Readiness probes tend to be written at the same level of ambition as the code they guard, so a checkout service can answer one correctly while returning 500s on the path customers actually use. That is how a clean sync and an all-green dashboard coexisted with paying customers getting errors ninety seconds later, according to the dev.to writeup [3].
Swapping in Rollouts does not change what ships, it changes who holds the opinion. The pod spec is untouched: kind: Deployment becomes kind: Rollout under argoproj.io/v1alpha1, and spec.strategy grows a canary block with explicit steps [6]. Argo CD goes back to reconciling Git, and the verdict moves to the canary controller [5].
The interesting failure is the next one, when two controllers both believe they own spec.replicas. With syncPolicy.automated.selfHeal set to true, a paused canary's reduced replica count reads as drift, and Argo CD corrects it; the author reports the canary snapping back to full replicas before the pause had finished [13]. The documented remedies are ignoreDifferences scoped to the Rollout kind, or switching selfHeal off during release windows [12].
Read the published spec as arithmetic and one more thing falls out. It runs six replicas [8] and opens at setWeight: 10 [7]. A single pod is 16.7 percent of six, so a 10 percent step has no representation in pod counts at all [1]. It means something only if Istio, an NGINX canary annotation or a Gateway API route is genuinely splitting by weight [11], which is why the traffic layer sits in the prerequisites rather than in a later step.
The step list is also thinner than it first looks. It contains exactly one analysis gate, placed before setWeight: 50, after which a three minute pause carries the release to full weight [3]. The explicit pauses add up to five minutes [2]. Set against a bad image reaching every pod in one sync cycle [4], five minutes and a single Prometheus verdict is a real improvement and a short leash: the deepest state anyone checked against SLIs is half the fleet, and the rest arrives on a clock.
Ranked by verification strength, evidence, and original report placement.
The team pushed a checkout service change, CI passed, Argo CD synced clean and the dashboard was all green; ninety seconds later their alerting channel filled with 500s from real customers in the checkout flow.
The replacement setup swaps the Deployment for an Argo Rollouts canary that ships to a small traffic slice first, gated by Prometheus SLIs for error rate and p99 latency, leaving Argo CD to keep the cluster in sync with Git.
A pause step on its own only buys time and still requires someone to decide whether to promote; an AnalysisTemplate turns the pause into an automated go/no-go decision based on actual metrics rather than a person watching a dashboard.
Argo CD reported the deploy as "Healthy", which meant only that the manifests had applied, not that the checkout service had stopped returning 500s.
Argo CD's health check for a plain Kubernetes Deployment only confirms that pods are running and ready; it has no opinion on whether the application returns correct responses.
With auto-sync on, no canary step and no automated rollback, the bad image went from zero to 100 percent of pods in a single sync cycle, and every replica was serving the broken build before a human noticed the error rate spike.
Follow any of these and your For You feed starts watching them — no settings page required.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Manifests verifiable, incident is unverified self-report
The technical substance - the Rollout kind change, the six-step canary list, the Prometheus AnalysisTemplate thresholds, the selfHeal drift conflict - is stated concretely and is internally checkable against the published YAML, which lifts evidence above anecdote level. But the cluster has exactly one source, syndicated from the author's own blog, and the triggering outage is asserted with no error-rate graph, postmortem, or timestamped record; the supplied body is also truncated mid-sentence, so later caveats cannot be checked. Nothing external corroborates either the incident or the post-migration result.
One anonymous team, no outcome data
The only adoption signal is a single self-disclosed migration of one checkout service to an Argo Rollouts canary with Prometheus gating. There is no organization name, no count of services or clusters, no time-in-production, and no post-change error-rate or MTTR figure. The Argo Rollouts controller and Prometheus are named as prerequisites, but the source discloses no usage beyond the author's own setup, so nothing broader can be inferred.
Framing slightly outruns the shipped spec
The headline claim about Argo CD health semantics is accurate and modestly stated, and the article itself lists prerequisites and gotchas rather than promising a silver bullet, which keeps the gap small. It tips positive because the promised 'gated by real SLIs before it's allowed to progress further' overstates the published spec: only one analysis gate exists and the jump from 50 to 100 percent of traffic is gated by a bare three-minute pause, the 10 percent weight is unattainable without a routing layer that is treated as a checklist item, and the claimed reliability benefit is asserted with no post-migration measurement.
Personal-brand syndication, no vendor tie disclosed
The article is republished from the author's own domain onto a developer platform, which creates an audience- and personal-reputation incentive to frame a routine misconfiguration as a dramatic near-miss. Offsetting that, the tooling recommended is open-source Argo project software, alternative metric providers (Prometheus, Datadog, CloudWatch) and routing layers are presented as interchangeable, and no sponsorship, employer, product, or affiliate relationship is claimed anywhere in the supplied text, so no commercial pull is evidenced.
High on mechanism, low on outcome
Confidence is bounded by single-publisher, single-author sourcing and a truncated body. The configuration-level claims can be trusted fairly well because they are shown as manifests and describe well-scoped controller behavior; the incident, the causal attribution, and any claim about improved reliability after the migration cannot be independently checked, and adoption breadth is effectively unknown.
build
An AI ops agent's real permissions design is two Istio policies and one ClusterRole1 distinct publisher
build
A NetworkPolicy in another repo broke invoicing while every dashboard reported success1 distinct publisher
build
Flux moves GitOps' source of truth into registries you own, and mirroring becomes the prerequisite1 distinct publisher
build
Thirteen tasks green, then "give up (Recommended)" on the one that needed understanding1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 25, 2026