Product1 publisher3 min readPublished
CNCF's Kubernetes recovery lab tests backups against a byte count and four known rows
The guidance runs three failure scenarios on a laptop, each one breaking a join between declared state and stored data, and its verification step is a number from Velero's data mover rather than a Completed status.
The Product Desk · Product desk

What happened
- CNCF published guidance built on three Kubernetes failure scenarios that separate having backups from being able to recover, each reproducible on a laptop from a lab repository with real terminal captures.
- The first scenario queries Velero's DataUpload objects rather than the backup status, and the capture shows 47,989,888 bytes of volume data leaving the cluster for the external object store.
- The document draws a boundary at the cluster itself: backup tools restore into a cluster that already exists and do not create nodes, networks, load balancers or DNS.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- decision Anyone who signs off DR readiness now has to decide whether a Completed backup phase is evidence at all, given the guidance holds that only an end to end recovery test shows the application starts and serves the expected data.
- constraint A runbook that opens at the restore step has no owner for the layer beneath it, so the infrastructure as code or Cluster API work becomes a prerequisite rather than a follow-up task.
- exposure Teams protecting a database with volume snapshots alone carry the risk of an inconsistent restore themselves, because the consistency depends on flush or quiesce hooks nobody configures by default.
- capability Because the scenarios run on two local clusters, a restore rehearsal becomes something one engineer can do in an afternoon instead of a change window negotiated with everyone who owns production.
47,989,888 bytes, about 45.8 MiB of volume data, is the number CNCF's guidance asks you to look up before believing anything else about a backup [14][23]. It comes out of Velero's DataUpload objects, not out of the backup's status field, because the status field reports that the backup operation finished and nothing about what left the cluster [12][13]. A tool that cannot produce that figure for a given run, the document says, deserves scrutiny [15]. The story a team tells itself is that the green backup list is recovery evidence, while what the list documents is a copy operation that ended without error [19].
The four-row Postgres does the other half of the work. Known contents mean a restore is checked against an expected result rather than against a dashboard [11]. Delete the namespace with its PVC, restore, and the same four rows come back in roughly two minutes [16]. Reading those two minutes as an RTO would repeat the error the document is built to expose, because it is a laptop restoring four rows, and the guidance is explicit that a completed operation does not prove an application starts, holds the right data, or serves traffic [16][19].
The structural claim is that each of the four recovery layers has mature tooling and usually comes back fine on its own, so the failures collect at the joins between them [5][6]. Scenario 2 is the one most GitOps shops will recognise: production is powered off, the recovery cluster's controller rebuilds the declarations from Git, and the store is where the data actually lives [22]. Every controller reports success over an empty volume, which is exactly the join the document names as an application definition that provisions nothing [6][22].
The happy path hides two more issues. Protecting volume data does not make a database backup application-consistent unless flush or quiesce hooks are configured for it [17]. Restoring onto different infrastructure needs storage class mappings and other transformations that the tool exposes but the team has to design and test [18]. And the boundary is drawn where most runbooks are vaguest: backup tools restore into a cluster that already exists, and the cluster, nodes, network, load balancers and DNS belong to infrastructure as code or Cluster API [20].
The material does not measure how often real teams fall into these joins, and it skips incident data, product comparison, and compliance mapping entirely, offering instead three mechanisms you can reproduce and real terminal captures from the lab that produced them [1][3].
The usable test is two yes/no questions per stateful namespace: whether your backup tool can report bytes moved for last night's run, and whether anyone can state the expected result of a restore precisely enough to check it. Yes on both is evidence. Bytes without an expected result only tells you a shipment arrived, not that it was the right cargo, while an expected result without a bytes figure lets you validate a restore but not show the backup carried data in the first place. Neither one is the dashboard that most teams are actually signing off on [13]. The cheapest way to move a quadrant is the one CNCF ran locally, and it costs a deleted namespace and a row count somebody wrote down in advance [11][16].
What to watch
- Whether backup vendors start surfacing a bytes-moved figure in the default view instead of a Completed badge.
- Whether the multi-volume consistency scenario lands as a reproducible test teams can run against their own databases.
- Whether DR runbooks begin naming an owner for the cluster layer, in infrastructure as code or Cluster API, rather than opening at the restore step.