A NetworkPolicy in another repo broke invoicing while every dashboard reported success

Each of those dashboards was accurate about the thing it was wired to watch, which is precisely why none of them helped. A deployment status attests that a rollout finished. It says nothing about whether the process it started can reach anything, and the author's point is that a healthy pod and an unrenderable invoice were true at the same moment ^[2]. No gate in the promotion path was asking the question a customer asks.

The part of the fix worth reading is not the YAML, it is the choice of which fields the pipeline enforces. Two do work today: the named NetworkPolicy must exist in the target cluster, and the smoke test runs after the migration ^[10]. The same file also declares the dependency's port 8080, a minimum dependency version of 4.12.0, and the flag `invoice_tax_v2` ^[9], and none of those are checked ^[19]. So the file is a contract in two places and a memo everywhere else, and memos decay the way the Confluence runbook did, last edited in February while still nominally the release procedure ^[8].

The write-up is one team's account of its own outage ^[20], and its most portable number is about noise. By the time the three partial views sat in one incident channel, 63 deployment messages had buried the one that mattered ^[5]: about one useful message in 64, or 1.6% of the traffic in the channel people were watching ^[16].

The reason no single team could see it is structural. Release contracts lived in five separate systems: image and Helm values in the service repo, network rules in the infrastructure repo, migrations in a separate GitHub Actions workflow, flags in LaunchDarkly owned by whoever last touched the feature, and the runbook in Confluence ^[8][17]. Four teams completed valid local work and nobody owned the seam between them ^[6]. Meanwhile the estate needing coordination went from six services and one cluster to 34 services and three production clusters, roughly 5.7 times the services ^[11][18], with the pager often held by someone who knows the application better than the person who built the cluster ^[12].

The access changes point at the same shape. On-call roles are namespace-scoped, reading pods, logs and events and patching deployments, with `pods/exec` verbs deliberately left empty ^[13]; cluster-wide secret reads, production database writes and unrestricted exec need an incident ticket and a second human ^[14]. The stated reason is a May incident where a live edit inside a container was erased by the next rollout, costing forty minutes of rediscovery ^[15]. A change that lives outside the repository is invisible to the pipeline, whether it is a network rule in another team's repo or a file typed into a running container. That is the same failure twice, and only one of the two now has something standing in front of it.